Max level shown:
12025-04-30T06:26:09.830ZDEBGtest_disk_expunge (init_with_steps): Running step start_crdb
22025-04-30T06:26:09.831ZDEBGtest_disk_expunge: Starting CRDB
32025-04-30T06:26:09.864ZINFOtest_disk_expunge: cockroach temporary directory: /var/tmp/omicron_tmp/.tmpWa1NgR
42025-04-30T06:26:09.864ZINFOtest_disk_expunge: cockroach: copying from seed tarball (/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar) to storage directory (/var/tmp/omicron_tmp/.tmpWa1NgR/data)
52025-04-30T06:26:09.883ZINFOtest_disk_expunge: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/var/tmp/omicron_tmp/.tmpWa1NgR/data,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /var/tmp/omicron_tmp/.tmpWa1NgR/listen-url
62025-04-30T06:26:09.884ZINFOtest_disk_expunge: cockroach environment: BUILDOMAT_JOB_ID=01JT2JWKGHF3JHFMW0JR13NSA6 BUILDOMAT_TASK_ID=4 CARGO=/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/build/.cargo CARGO_INCREMENTAL=0 CARGO_MANIFEST_DIR=/work/oxidecomputer/omicron/nexus CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=omicron-nexus CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CARGO_TERM_COLOR=always CI=true CRDB_SEED_TAR=/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar GITHUB_REPOSITORY=oxidecomputer/omicron GITHUB_SHA=531912b3fc900900bc7bd18f5eac1ce1e22cb1cd GOTRACEBACK=crash HOME=/home/build LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib LOGNAME=build NEXTEST=1 NEXTEST_BIN_EXE_nexus=/work/oxidecomputer/omicron/target/debug/nexus NEXTEST_BIN_EXE_schema-updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_EXECUTION_MODE=process-per-test NEXTEST_LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib NEXTEST_PROFILE=ci NEXTEST_RUN_ID=29735cc2-0883-4059-8ca2-1da53f01ede3 NEXTEST_TEST_GLOBAL_SLOT=6 NEXTEST_TEST_GROUP=@global NEXTEST_TEST_GROUP_SLOT=none OUT_DIR=/work/oxidecomputer/omicron/target/debug/build/omicron-nexus-553062f859dbbfa0/out PATH=/work/oxidecomputer/omicron/out/mgd/root/opt/oxide/mgd/bin:/work/oxidecomputer/omicron/out/dendrite-stub/bin:/work/oxidecomputer/omicron/out/clickhouse:/work/oxidecomputer/omicron/out/cockroachdb/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin PWD=/work/oxidecomputer/omicron RUSTC_BOOTSTRAP=1 RUSTDOCFLAGS=--document-private-items -D warnings RUSTFLAGS=-D warnings -C link-arg=-R/usr/platform/oxide/lib/amd64 RUSTUP_HOME=/home/build/.rustup RUSTUP_TOOLCHAIN=1.86.0-x86_64-unknown-illumos RUST_BACKTRACE=1 RUST_RECURSION_COUNT=1 SHLVL=1 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TMPDIR=/var/tmp/omicron_tmp TZ=UTC USER=build _=/usr/bin/ptime __NEXTEST_ATTEMPT=1
72025-04-30T06:26:11.333ZINFOtest_disk_expunge: cockroach pid: 21925
82025-04-30T06:26:11.333ZINFOtest_disk_expunge: cockroach listen URL: postgresql://root@[::1]:63866/omicron?sslmode=disable
92025-04-30T06:26:11.333ZDEBGtest_disk_expunge (init_with_steps): Running step start_clickhouse
102025-04-30T06:26:11.333ZDEBGtest_disk_expunge: Starting Clickhouse
112025-04-30T06:26:11.932ZDEBGtest_disk_expunge (init_with_steps): Running step start_gateway_switch0
122025-04-30T06:26:11.932ZDEBGtest_disk_expunge: Starting Management Gateway
132025-04-30T06:26:12.055ZDEBGtest_disk_expunge (init_with_steps): Running step start_dendrite_switch0
142025-04-30T06:26:12.056ZDEBGtest_disk_expunge: Starting Dendrite for switch0
152025-04-30T06:26:12.101ZDEBGtest_disk_expunge (init_with_steps): Running step start_mgd_switch0
162025-04-30T06:26:12.101ZDEBGtest_disk_expunge: Starting mgd for switch0
172025-04-30T06:26:12.251ZDEBGtest_disk_expunge: mgd port is 61919
182025-04-30T06:26:12.251ZDEBGtest_disk_expunge (init_with_steps): Running step record_switch_dns
192025-04-30T06:26:12.251ZDEBGtest_disk_expunge: Recording DNS for the switch zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 switch_location = switch0
202025-04-30T06:26:12.251ZDEBGtest_disk_expunge (init_with_steps): Running step start_internal_dns
212025-04-30T06:26:12.252ZINFOtest_disk_expunge (store): opening sled database path = /var/tmp/omicron_tmp/.tmpTLiaxG
222025-04-30T06:26:12.443ZINFOtest_disk_expunge (store): pruning trees for generations newer than 0
232025-04-30T06:26:12.443ZINFOtest_disk_expunge (store): pruning trees for generations older than 0 keep = 3
242025-04-30T06:26:12.443ZINFOtest_disk_expunge (dns): DNS server bound to address kind = dns local_address = [::1]:58353
252025-04-30T06:26:12.445ZDEBGtest_disk_expunge (http): registered endpoint kind = dns local_addr = [::1]:36493 method = GET path = /config versions = all
262025-04-30T06:26:12.445ZDEBGtest_disk_expunge (http): registered endpoint kind = dns local_addr = [::1]:36493 method = PUT path = /config versions = all
272025-04-30T06:26:12.445ZINFOtest_disk_expunge (http): listening kind = dns local_addr = [::1]:36493
282025-04-30T06:26:12.446ZDEBGtest_disk_expunge (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:36493
292025-04-30T06:26:12.446ZDEBGtest_disk_expunge (init_with_steps): Running step start_external_dns
302025-04-30T06:26:12.447ZINFOtest_disk_expunge (store): opening sled database path = /var/tmp/omicron_tmp/.tmpyNoXR1
312025-04-30T06:26:12.457ZINFOtest_disk_expunge (store): pruning trees for generations newer than 0
322025-04-30T06:26:12.457ZINFOtest_disk_expunge (store): pruning trees for generations older than 0 keep = 3
332025-04-30T06:26:12.457ZINFOtest_disk_expunge (dns): DNS server bound to address kind = dns local_address = [::1]:61245
342025-04-30T06:26:12.458ZDEBGtest_disk_expunge (http): registered endpoint kind = dns local_addr = [::1]:52024 method = GET path = /config versions = all
352025-04-30T06:26:12.458ZDEBGtest_disk_expunge (http): registered endpoint kind = dns local_addr = [::1]:52024 method = PUT path = /config versions = all
362025-04-30T06:26:12.458ZINFOtest_disk_expunge (http): listening kind = dns local_addr = [::1]:52024
372025-04-30T06:26:12.458ZDEBGtest_disk_expunge (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:52024
382025-04-30T06:26:12.459ZDEBGtest_disk_expunge (init_with_steps): Running step start_nexus_internal
392025-04-30T06:26:12.459ZDEBGtest_disk_expunge: Starting Nexus (internal API)
402025-04-30T06:26:12.460ZINFOtest_disk_expunge: setting up nexus server
412025-04-30T06:26:12.462ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Action
422025-04-30T06:26:12.462ZINFOtest_disk_expunge (ServerContext): registering Oso class class = AnyActor
432025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = AuthenticatedActor
442025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = BlueprintConfig
452025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Database
462025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = DnsConfig
472025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Fleet
482025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Inventory
492025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = IpPoolList
502025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = ConsoleSessionList
512025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = DeviceAuthRequestList
522025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = SiloCertificateList
532025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = SiloIdentityProviderList
542025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = SiloUserList
552025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = TargetReleaseConfig
562025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = WebhookEventClassList
572025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Project
582025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Disk
592025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Snapshot
602025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = ProjectImage
612025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = AffinityGroup
622025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = AntiAffinityGroup
632025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Instance
642025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = IpPool
652025-04-30T06:26:12.463ZINFOtest_disk_expunge (ServerContext): registering Oso class class = InstanceNetworkInterface
662025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Vpc
672025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = VpcRouter
682025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = InternetGateway
692025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = InternetGatewayIpPool
702025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = InternetGatewayIpAddress
712025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = RouterRoute
722025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = VpcSubnet
732025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = FloatingIp
742025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Image
752025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = SiloImage
762025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = AddressLot
772025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Blueprint
782025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = LoopbackAddress
792025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Certificate
802025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = ConsoleSession
812025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = DeviceAuthRequest
822025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = DeviceAccessToken
832025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = PhysicalDisk
842025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Rack
852025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = RoleBuiltin
862025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = SshKey
872025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Silo
882025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = SiloUser
892025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = SiloGroup
902025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = SupportBundle
912025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = IdentityProvider
922025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = SamlIdentityProvider
932025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Sled
942025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = TufRepo
952025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = TufArtifact
962025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = WebhookEvent
972025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = WebhookReceiver
982025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = WebhookSecret
992025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Zpool
1002025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = Service
1012025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): registering Oso class class = UserBuiltin
1022025-04-30T06:26:12.464ZINFOtest_disk_expunge (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
1032025-04-30T06:26:12.500ZINFOtest_disk_expunge (ServerContext): Setting up resolver using DNS address: [::1]:58353
1042025-04-30T06:26:12.501ZINFOtest_disk_expunge (DnsResolver): new DNS resolver addresses = [[::1]:58353]
1052025-04-30T06:26:12.501ZINFOtest_disk_expunge (ServerContext): Setting up qorb database pool from a single host url = PostgresConfigWithUrl {\n url_raw: "postgresql://root@[::1]:63866/omicron?sslmode=disable",\n config: Config {\n user: Some(\n "root",\n ),\n password: None,\n dbname: Some(\n "omicron",\n ),\n options: None,\n application_name: None,\n ssl_mode: Disable,\n host: [\n Tcp(\n "::1",\n ),\n ],\n hostaddr: [],\n port: [\n 63866,\n ],\n connect_timeout: None,\n tcp_user_timeout: None,\n keepalives: true,\n keepalives_idle: 7200s,\n keepalives_interval: None,\n keepalives_retries: None,\n target_session_attrs: Any,\n channel_binding: Prefer,\n load_balance_hosts: Disable,\n },\n}
1062025-04-30T06:26:12.503ZDEBGtest_disk_expunge (ServerContext): registered USDT probes
1072025-04-30T06:26:12.580ZINFOtest_disk_expunge (nexus): Database schema version is up to date desired_version = 140.0.0 found_version = 140.0.0
1082025-04-30T06:26:12.765ZDEBGtest_disk_expunge (clickhouse-client): registered USDT probes id = 92c9df21-a651-47f6-b92e-f954076e535b
1092025-04-30T06:26:12.869ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /bgtasks versions = all
1102025-04-30T06:26:12.869ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /bgtasks/activate versions = all
1112025-04-30T06:26:12.869ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /bgtasks/view/{bgtask_name} versions = all
1122025-04-30T06:26:12.870ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /clickhouse/policy versions = all
1132025-04-30T06:26:12.870ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /clickhouse/policy versions = all
1142025-04-30T06:26:12.870ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
1152025-04-30T06:26:12.870ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1162025-04-30T06:26:12.870ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1172025-04-30T06:26:12.870ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1182025-04-30T06:26:12.870ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1192025-04-30T06:26:12.870ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /demo-saga versions = all
1202025-04-30T06:26:12.871ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1212025-04-30T06:26:12.871ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /deployment/blueprints/all versions = all
1222025-04-30T06:26:12.871ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1232025-04-30T06:26:12.871ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1242025-04-30T06:26:12.871ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /deployment/blueprints/import versions = all
1252025-04-30T06:26:12.871ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /deployment/blueprints/regenerate versions = all
1262025-04-30T06:26:12.871ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /deployment/blueprints/target versions = all
1272025-04-30T06:26:12.871ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /deployment/blueprints/target versions = all
1282025-04-30T06:26:12.871ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = PUT path = /deployment/blueprints/target/enabled versions = all
1292025-04-30T06:26:12.871ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1302025-04-30T06:26:12.872ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = PUT path = /disks/{disk_id} versions = all
1312025-04-30T06:26:12.872ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /experimental/v1/system/support-bundles versions = all
1322025-04-30T06:26:12.872ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /experimental/v1/system/support-bundles versions = all
1332025-04-30T06:26:12.872ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1342025-04-30T06:26:12.872ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1352025-04-30T06:26:12.872ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1362025-04-30T06:26:12.872ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1372025-04-30T06:26:12.872ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1382025-04-30T06:26:12.872ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1392025-04-30T06:26:12.872ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
1402025-04-30T06:26:12.873ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /instances/{instance_id}/migrate versions = all
1412025-04-30T06:26:12.873ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /metrics/collectors versions = all
1422025-04-30T06:26:12.873ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1432025-04-30T06:26:12.873ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /metrics/producers versions = all
1442025-04-30T06:26:12.873ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /mgs-updates versions = all
1452025-04-30T06:26:12.873ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1462025-04-30T06:26:12.873ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /oximeter/read-policy versions = all
1472025-04-30T06:26:12.873ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /oximeter/read-policy versions = all
1482025-04-30T06:26:12.873ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /physical-disk/expunge versions = all
1492025-04-30T06:26:12.873ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /probes/{sled} versions = all
1502025-04-30T06:26:12.873ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1512025-04-30T06:26:12.874ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /sagas versions = all
1522025-04-30T06:26:12.874ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /sagas/{saga_id} versions = all
1532025-04-30T06:26:12.874ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /sled-agents/{sled_id} versions = all
1542025-04-30T06:26:12.874ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /sled-agents/{sled_id} versions = all
1552025-04-30T06:26:12.874ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1562025-04-30T06:26:12.874ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /sleds/add versions = all
1572025-04-30T06:26:12.874ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /sleds/expunge versions = all
1582025-04-30T06:26:12.874ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /sleds/uninitialized versions = all
1592025-04-30T06:26:12.874ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = PUT path = /switch/{switch_id} versions = all
1602025-04-30T06:26:12.874ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = GET path = /v1/ping versions = all
1612025-04-30T06:26:12.874ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = PUT path = /vmms/{propolis_id} versions = all
1622025-04-30T06:26:12.875ZDEBGtest_disk_expunge (dropshot_internal): registered endpoint local_addr = 127.0.0.1:34036 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1632025-04-30T06:26:12.875ZINFOtest_disk_expunge (dropshot_internal): listening local_addr = 127.0.0.1:34036
1642025-04-30T06:26:12.876ZDEBGtest_disk_expunge (dropshot_internal): successfully registered DTrace USDT probes local_addr = 127.0.0.1:34036
1652025-04-30T06:26:12.876ZINFOtest_disk_expunge (SEC): SEC running sec_id = 78121327-166d-4e33-935c-82ab8635bf33
1662025-04-30T06:26:12.878ZTRACtest_disk_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1672025-04-30T06:26:12.879ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1682025-04-30T06:26:12.882ZDEBGtest_disk_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1692025-04-30T06:26:12.882ZDEBGtest_disk_expunge (DataLoader): attempting to create built-in users
1702025-04-30T06:26:12.883ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1712025-04-30T06:26:12.883ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1722025-04-30T06:26:12.884ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1732025-04-30T06:26:12.884ZINFOtest_disk_expunge (MgsUpdateDriver): starting MgsUpdateDriver
1742025-04-30T06:26:12.917ZTRACtest_disk_expunge (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 8d 9d 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 ............._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:53826 req_id = 61eba8b7-d539-4b20-a51e-d5b612459785
1752025-04-30T06:26:12.921ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36253,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53826 req_id = 61eba8b7-d539-4b20-a51e-d5b612459785
1762025-04-30T06:26:12.922ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36253, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_crucible-pantry._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 16, 95, 99, 114, 117, 99, 105, 98, 108, 101, 45, 112, 97, 110, 116, 114, 121, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_crucible-pantry._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53826 req_id = 61eba8b7-d539-4b20-a51e-d5b612459785
1772025-04-30T06:26:12.981ZINFOtest_disk_expunge (DataLoader): created 5 built-in users
1782025-04-30T06:26:12.982ZTRACtest_disk_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1792025-04-30T06:26:12.983ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1802025-04-30T06:26:12.985ZDEBGtest_disk_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1812025-04-30T06:26:12.985ZDEBGtest_disk_expunge (DataLoader): attempting to create built-in roles
1822025-04-30T06:26:12.985ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1832025-04-30T06:26:12.985ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1842025-04-30T06:26:12.986ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1852025-04-30T06:26:13.082ZINFOtest_disk_expunge (DataLoader): created 10 built-in roles
1862025-04-30T06:26:13.083ZTRACtest_disk_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1872025-04-30T06:26:13.083ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1882025-04-30T06:26:13.086ZDEBGtest_disk_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1892025-04-30T06:26:13.086ZDEBGtest_disk_expunge (DataLoader): attempting to create built-in role assignments
1902025-04-30T06:26:13.086ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1912025-04-30T06:26:13.086ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1922025-04-30T06:26:13.087ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1932025-04-30T06:26:13.410ZINFOtest_disk_expunge (DataLoader): created 4 built-in role assignments
1942025-04-30T06:26:13.410ZTRACtest_disk_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1952025-04-30T06:26:13.410ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
1962025-04-30T06:26:13.412ZDEBGtest_disk_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1972025-04-30T06:26:13.412ZDEBGtest_disk_expunge (DataLoader): attempting to create built-in silos
1982025-04-30T06:26:13.412ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1992025-04-30T06:26:13.412ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2002025-04-30T06:26:13.412ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2012025-04-30T06:26:13.516ZINFOtest_disk_expunge (DataLoader): created 2 built-in silos
2022025-04-30T06:26:13.516ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2032025-04-30T06:26:13.516ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2042025-04-30T06:26:13.517ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2052025-04-30T06:26:13.535ZTRACtest_disk_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2062025-04-30T06:26:13.535ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2072025-04-30T06:26:13.537ZDEBGtest_disk_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2082025-04-30T06:26:13.537ZDEBGtest_disk_expunge (DataLoader): attempting to create built-in projects
2092025-04-30T06:26:13.537ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2102025-04-30T06:26:13.538ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2112025-04-30T06:26:13.538ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2122025-04-30T06:26:13.542ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2132025-04-30T06:26:13.543ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2142025-04-30T06:26:13.543ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2152025-04-30T06:26:13.543ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2162025-04-30T06:26:13.543ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2172025-04-30T06:26:13.587ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2182025-04-30T06:26:13.587ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2192025-04-30T06:26:13.587ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2202025-04-30T06:26:13.587ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2212025-04-30T06:26:13.641ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2222025-04-30T06:26:13.645ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2232025-04-30T06:26:13.645ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2242025-04-30T06:26:13.645ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2252025-04-30T06:26:13.645ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2262025-04-30T06:26:13.646ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2272025-04-30T06:26:13.646ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2282025-04-30T06:26:13.649ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2292025-04-30T06:26:13.649ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2302025-04-30T06:26:13.649ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2312025-04-30T06:26:13.650ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2322025-04-30T06:26:13.652ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2332025-04-30T06:26:13.656ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2342025-04-30T06:26:13.656ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2352025-04-30T06:26:13.656ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2362025-04-30T06:26:13.657ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2372025-04-30T06:26:13.680ZINFOtest_disk_expunge (DataLoader): created built-in services project
2382025-04-30T06:26:13.681ZTRACtest_disk_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2392025-04-30T06:26:13.681ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2402025-04-30T06:26:13.682ZDEBGtest_disk_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2412025-04-30T06:26:13.682ZDEBGtest_disk_expunge (DataLoader): attempting to create built-in VPCs
2422025-04-30T06:26:13.682ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2432025-04-30T06:26:13.682ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2442025-04-30T06:26:13.683ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2452025-04-30T06:26:13.686ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2462025-04-30T06:26:13.687ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2472025-04-30T06:26:13.687ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2482025-04-30T06:26:13.690ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2492025-04-30T06:26:13.690ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2502025-04-30T06:26:13.690ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2512025-04-30T06:26:13.690ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2522025-04-30T06:26:13.690ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2532025-04-30T06:26:13.694ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2542025-04-30T06:26:13.694ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2552025-04-30T06:26:13.694ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2562025-04-30T06:26:13.694ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-04-30T06:26:13.697ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2582025-04-30T06:26:13.697ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2592025-04-30T06:26:13.697ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2602025-04-30T06:26:13.697ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2612025-04-30T06:26:13.699ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2622025-04-30T06:26:13.704ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2632025-04-30T06:26:13.704ZTRACtest_disk_expunge (DataLoader): successful lookup of siloed resource "Project" using built-in user
2642025-04-30T06:26:13.704ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2652025-04-30T06:26:13.704ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2662025-04-30T06:26:13.704ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2672025-04-30T06:26:13.704ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2682025-04-30T06:26:13.705ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2692025-04-30T06:26:13.707ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2702025-04-30T06:26:13.707ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2712025-04-30T06:26:13.707ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2722025-04-30T06:26:13.708ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2732025-04-30T06:26:13.710ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2742025-04-30T06:26:13.710ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2752025-04-30T06:26:13.710ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2762025-04-30T06:26:13.710ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2772025-04-30T06:26:13.712ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2782025-04-30T06:26:13.717ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2792025-04-30T06:26:13.717ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2802025-04-30T06:26:13.717ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2812025-04-30T06:26:13.717ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2822025-04-30T06:26:13.730ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
2832025-04-30T06:26:13.730ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2842025-04-30T06:26:13.730ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2852025-04-30T06:26:13.730ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2862025-04-30T06:26:13.730ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2872025-04-30T06:26:13.733ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2882025-04-30T06:26:13.734ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2892025-04-30T06:26:13.734ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2902025-04-30T06:26:13.734ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2912025-04-30T06:26:13.736ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2922025-04-30T06:26:13.736ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2932025-04-30T06:26:13.736ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2942025-04-30T06:26:13.736ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2952025-04-30T06:26:13.739ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2962025-04-30T06:26:13.745ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
2972025-04-30T06:26:13.745ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2982025-04-30T06:26:13.745ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
2992025-04-30T06:26:13.746ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3002025-04-30T06:26:13.752ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3012025-04-30T06:26:13.752ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3022025-04-30T06:26:13.753ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3032025-04-30T06:26:13.762ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3042025-04-30T06:26:13.762ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3052025-04-30T06:26:13.762ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3062025-04-30T06:26:13.768ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3072025-04-30T06:26:13.768ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3082025-04-30T06:26:13.768ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3092025-04-30T06:26:13.768ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3102025-04-30T06:26:13.769ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3112025-04-30T06:26:13.772ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3122025-04-30T06:26:13.772ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3132025-04-30T06:26:13.772ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3142025-04-30T06:26:13.773ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-04-30T06:26:13.776ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3162025-04-30T06:26:13.776ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3172025-04-30T06:26:13.776ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3182025-04-30T06:26:13.778ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3192025-04-30T06:26:13.780ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3202025-04-30T06:26:13.787ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3212025-04-30T06:26:13.787ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3222025-04-30T06:26:13.787ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3232025-04-30T06:26:13.787ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3242025-04-30T06:26:13.826ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3252025-04-30T06:26:13.826ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3262025-04-30T06:26:13.826ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3272025-04-30T06:26:13.826ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3282025-04-30T06:26:13.826ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3292025-04-30T06:26:13.829ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3302025-04-30T06:26:13.829ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3312025-04-30T06:26:13.829ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3322025-04-30T06:26:13.830ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-04-30T06:26:13.833ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3342025-04-30T06:26:13.833ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3352025-04-30T06:26:13.833ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3362025-04-30T06:26:13.834ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3372025-04-30T06:26:13.837ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3382025-04-30T06:26:13.847ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3392025-04-30T06:26:13.847ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3402025-04-30T06:26:13.847ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3412025-04-30T06:26:13.847ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3422025-04-30T06:26:13.929ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3432025-04-30T06:26:13.929ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3442025-04-30T06:26:13.929ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3452025-04-30T06:26:13.929ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3462025-04-30T06:26:13.930ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3472025-04-30T06:26:13.933ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3482025-04-30T06:26:13.933ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3492025-04-30T06:26:13.933ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3502025-04-30T06:26:13.933ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-04-30T06:26:13.937ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3522025-04-30T06:26:13.937ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3532025-04-30T06:26:13.937ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3542025-04-30T06:26:13.939ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3552025-04-30T06:26:13.946ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3562025-04-30T06:26:13.956ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3572025-04-30T06:26:13.956ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3582025-04-30T06:26:13.956ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3592025-04-30T06:26:13.957ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3602025-04-30T06:26:13.967ZDEBGtest_disk_expunge (DataLoader): attempting to create built-in VPC firewall rules
3612025-04-30T06:26:13.968ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3622025-04-30T06:26:13.968ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3632025-04-30T06:26:13.969ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3642025-04-30T06:26:13.975ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3652025-04-30T06:26:13.975ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3662025-04-30T06:26:13.978ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3672025-04-30T06:26:13.985ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3682025-04-30T06:26:13.985ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3692025-04-30T06:26:13.986ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3702025-04-30T06:26:13.990ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3712025-04-30T06:26:13.990ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3722025-04-30T06:26:13.990ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3732025-04-30T06:26:13.990ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3742025-04-30T06:26:13.991ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3752025-04-30T06:26:13.994ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3762025-04-30T06:26:13.994ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3772025-04-30T06:26:13.994ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3782025-04-30T06:26:13.995ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3792025-04-30T06:26:13.997ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3802025-04-30T06:26:13.997ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3812025-04-30T06:26:13.997ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3822025-04-30T06:26:13.998ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3832025-04-30T06:26:14.001ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3842025-04-30T06:26:14.008ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3852025-04-30T06:26:14.008ZTRACtest_disk_expunge (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
3862025-04-30T06:26:14.008ZTRACtest_disk_expunge (DataLoader): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3872025-04-30T06:26:14.008ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3882025-04-30T06:26:14.008ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3892025-04-30T06:26:14.008ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3902025-04-30T06:26:14.009ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3912025-04-30T06:26:14.011ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3922025-04-30T06:26:14.011ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3932025-04-30T06:26:14.011ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3942025-04-30T06:26:14.011ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3952025-04-30T06:26:14.013ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3962025-04-30T06:26:14.014ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3972025-04-30T06:26:14.014ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
3982025-04-30T06:26:14.015ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3992025-04-30T06:26:14.026ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4002025-04-30T06:26:14.036ZDEBGtest_disk_expunge (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4012025-04-30T06:26:14.036ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4022025-04-30T06:26:14.036ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4032025-04-30T06:26:14.037ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4042025-04-30T06:26:14.054ZTRACtest_disk_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4052025-04-30T06:26:14.054ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4062025-04-30T06:26:14.054ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4072025-04-30T06:26:14.054ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4082025-04-30T06:26:14.055ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4092025-04-30T06:26:14.060ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4102025-04-30T06:26:14.060ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4112025-04-30T06:26:14.060ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4122025-04-30T06:26:14.060ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4132025-04-30T06:26:14.069ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4142025-04-30T06:26:14.070ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4152025-04-30T06:26:14.070ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4162025-04-30T06:26:14.071ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4172025-04-30T06:26:14.078ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4182025-04-30T06:26:14.087ZDEBGtest_disk_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4192025-04-30T06:26:14.087ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4202025-04-30T06:26:14.087ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4212025-04-30T06:26:14.088ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4222025-04-30T06:26:14.185ZINFOtest_disk_expunge (DataLoader): created built-in VPC firewall rules
4232025-04-30T06:26:14.185ZDEBGtest_disk_expunge (DataLoader): attempting to create built-in VPC Subnets
4242025-04-30T06:26:14.185ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4252025-04-30T06:26:14.186ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4262025-04-30T06:26:14.187ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4272025-04-30T06:26:14.217ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4282025-04-30T06:26:14.230ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4292025-04-30T06:26:14.233ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4302025-04-30T06:26:14.237ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4312025-04-30T06:26:14.240ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4322025-04-30T06:26:14.241ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4332025-04-30T06:26:14.244ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4342025-04-30T06:26:14.244ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4352025-04-30T06:26:14.244ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4362025-04-30T06:26:14.244ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4372025-04-30T06:26:14.245ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4382025-04-30T06:26:14.249ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4392025-04-30T06:26:14.249ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4402025-04-30T06:26:14.249ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4412025-04-30T06:26:14.250ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4422025-04-30T06:26:14.253ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4432025-04-30T06:26:14.253ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4442025-04-30T06:26:14.253ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4452025-04-30T06:26:14.255ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4462025-04-30T06:26:14.260ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4472025-04-30T06:26:14.269ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4482025-04-30T06:26:14.269ZTRACtest_disk_expunge (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4492025-04-30T06:26:14.269ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4502025-04-30T06:26:14.269ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4512025-04-30T06:26:14.272ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4522025-04-30T06:26:14.278ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4532025-04-30T06:26:14.278ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4542025-04-30T06:26:14.278ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4552025-04-30T06:26:14.278ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4562025-04-30T06:26:14.278ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4572025-04-30T06:26:14.283ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4582025-04-30T06:26:14.283ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4592025-04-30T06:26:14.283ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4602025-04-30T06:26:14.284ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4612025-04-30T06:26:14.289ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4622025-04-30T06:26:14.289ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4632025-04-30T06:26:14.289ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4642025-04-30T06:26:14.290ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4652025-04-30T06:26:14.292ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4662025-04-30T06:26:14.299ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4672025-04-30T06:26:14.306ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
4682025-04-30T06:26:14.306ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4692025-04-30T06:26:14.306ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4702025-04-30T06:26:14.306ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4712025-04-30T06:26:14.306ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4722025-04-30T06:26:14.308ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4732025-04-30T06:26:14.308ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4742025-04-30T06:26:14.308ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4752025-04-30T06:26:14.309ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4762025-04-30T06:26:14.311ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4772025-04-30T06:26:14.311ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4782025-04-30T06:26:14.311ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4792025-04-30T06:26:14.312ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4802025-04-30T06:26:14.315ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4812025-04-30T06:26:14.322ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
4822025-04-30T06:26:14.322ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4832025-04-30T06:26:14.322ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4842025-04-30T06:26:14.323ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4852025-04-30T06:26:14.388ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4862025-04-30T06:26:14.388ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4872025-04-30T06:26:14.389ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4882025-04-30T06:26:14.395ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4892025-04-30T06:26:14.395ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4902025-04-30T06:26:14.395ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4912025-04-30T06:26:14.395ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4922025-04-30T06:26:14.396ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4932025-04-30T06:26:14.401ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4942025-04-30T06:26:14.401ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4952025-04-30T06:26:14.401ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
4962025-04-30T06:26:14.402ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4972025-04-30T06:26:14.405ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4982025-04-30T06:26:14.405ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4992025-04-30T06:26:14.405ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5002025-04-30T06:26:14.406ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5012025-04-30T06:26:14.409ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5022025-04-30T06:26:14.427ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5032025-04-30T06:26:14.437ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5042025-04-30T06:26:14.438ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5052025-04-30T06:26:14.438ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5062025-04-30T06:26:14.438ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5072025-04-30T06:26:14.439ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5082025-04-30T06:26:14.442ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5092025-04-30T06:26:14.442ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5102025-04-30T06:26:14.442ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5112025-04-30T06:26:14.443ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5122025-04-30T06:26:14.446ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5132025-04-30T06:26:14.446ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5142025-04-30T06:26:14.446ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5152025-04-30T06:26:14.446ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5162025-04-30T06:26:14.449ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5172025-04-30T06:26:14.456ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5182025-04-30T06:26:14.456ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5192025-04-30T06:26:14.456ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5202025-04-30T06:26:14.456ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5212025-04-30T06:26:14.463ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5222025-04-30T06:26:14.463ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5232025-04-30T06:26:14.464ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5242025-04-30T06:26:14.465ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5252025-04-30T06:26:14.466ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5262025-04-30T06:26:14.466ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5272025-04-30T06:26:14.466ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5282025-04-30T06:26:14.466ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5292025-04-30T06:26:14.468ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5302025-04-30T06:26:14.468ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5312025-04-30T06:26:14.468ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5322025-04-30T06:26:14.469ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5332025-04-30T06:26:14.471ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5342025-04-30T06:26:14.471ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5352025-04-30T06:26:14.471ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5362025-04-30T06:26:14.472ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5372025-04-30T06:26:14.474ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5382025-04-30T06:26:14.479ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5392025-04-30T06:26:14.483ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5402025-04-30T06:26:14.483ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5412025-04-30T06:26:14.483ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5422025-04-30T06:26:14.483ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5432025-04-30T06:26:14.484ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5442025-04-30T06:26:14.486ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5452025-04-30T06:26:14.486ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5462025-04-30T06:26:14.486ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5472025-04-30T06:26:14.486ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5482025-04-30T06:26:14.488ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5492025-04-30T06:26:14.489ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5502025-04-30T06:26:14.489ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5512025-04-30T06:26:14.489ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5522025-04-30T06:26:14.491ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5532025-04-30T06:26:14.496ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5542025-04-30T06:26:14.496ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5552025-04-30T06:26:14.496ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5562025-04-30T06:26:14.497ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5572025-04-30T06:26:14.503ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5582025-04-30T06:26:14.503ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5592025-04-30T06:26:14.504ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5602025-04-30T06:26:14.508ZINFOtest_disk_expunge (DataLoader): created built-in services vpc subnets
5612025-04-30T06:26:14.508ZINFOtest_disk_expunge (DataLoader): created built-in services vpc
5622025-04-30T06:26:14.508ZTRACtest_disk_expunge (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5632025-04-30T06:26:14.508ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5642025-04-30T06:26:14.512ZDEBGtest_disk_expunge (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5652025-04-30T06:26:14.512ZDEBGtest_disk_expunge (DataLoader): attempting to create silo users
5662025-04-30T06:26:14.512ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5672025-04-30T06:26:14.512ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5682025-04-30T06:26:14.513ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5692025-04-30T06:26:14.520ZINFOtest_disk_expunge (DataLoader): created 2 silo users
5702025-04-30T06:26:14.520ZDEBGtest_disk_expunge (DataLoader): attempting to create silo user role assignments
5712025-04-30T06:26:14.520ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5722025-04-30T06:26:14.520ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5732025-04-30T06:26:14.521ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5742025-04-30T06:26:14.533ZINFOtest_disk_expunge (DataLoader): created 2 silo user role assignments
5752025-04-30T06:26:14.534ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5762025-04-30T06:26:14.534ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5772025-04-30T06:26:14.535ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5782025-04-30T06:26:14.548ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5792025-04-30T06:26:14.548ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5802025-04-30T06:26:14.549ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5812025-04-30T06:26:14.560ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5822025-04-30T06:26:14.560ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5832025-04-30T06:26:14.560ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5842025-04-30T06:26:14.560ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5852025-04-30T06:26:14.561ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5862025-04-30T06:26:14.565ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5872025-04-30T06:26:14.567ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5882025-04-30T06:26:14.568ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5892025-04-30T06:26:14.568ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5902025-04-30T06:26:14.568ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5912025-04-30T06:26:14.575ZTRACtest_disk_expunge (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5922025-04-30T06:26:14.576ZTRACtest_disk_expunge (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5932025-04-30T06:26:14.576ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5942025-04-30T06:26:14.576ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5952025-04-30T06:26:14.577ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5962025-04-30T06:26:14.579ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
5972025-04-30T06:26:14.581ZDEBGtest_disk_expunge (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5982025-04-30T06:26:14.581ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5992025-04-30T06:26:14.581ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
6002025-04-30T06:26:14.582ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6012025-04-30T06:26:14.657ZTRACtest_disk_expunge (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6022025-04-30T06:26:14.657ZDEBGtest_disk_expunge (DataLoader): roles roles = RoleSet { roles: {} }
6032025-04-30T06:26:14.659ZDEBGtest_disk_expunge (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6042025-04-30T06:26:14.664ZINFOtest_disk_expunge (nexus): populate complete
6052025-04-30T06:26:14.664ZINFOtest_disk_expunge (nexus): activating background tasks
6062025-04-30T06:26:14.668ZDEBGtest_disk_expunge (init_with_steps): Running step start_sled1
6072025-04-30T06:26:14.668ZINFOtest_disk_expunge (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6082025-04-30T06:26:14.728ZINFOtest_disk_expunge (SledAgent): created simulated sled agent server = b6d65341-167c-41df-9b5c-41cded99c229 sim_mode = Explicit sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6092025-04-30T06:26:14.728ZINFOtest_disk_expunge (omicron_sled_agent::sim::SimulatedUpstairs): registered sled b6d65341-167c-41df-9b5c-41cded99c229 storage
6102025-04-30T06:26:14.728ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
6112025-04-30T06:26:14.728ZDEBGtest_disk_expunge (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6122025-04-30T06:26:14.728ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
6132025-04-30T06:26:14.728ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6142025-04-30T06:26:14.729ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6152025-04-30T06:26:14.729ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
6162025-04-30T06:26:14.729ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
6172025-04-30T06:26:14.730ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
6182025-04-30T06:26:14.730ZWARNtest_disk_expunge (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config nor servers
6192025-04-30T06:26:14.730ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 99.558\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6202025-04-30T06:26:14.730ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
6212025-04-30T06:26:14.730ZDEBGtest_disk_expunge (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
6222025-04-30T06:26:14.730ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
6232025-04-30T06:26:14.730ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6242025-04-30T06:26:14.730ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6252025-04-30T06:26:14.730ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
6262025-04-30T06:26:14.730ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
6272025-04-30T06:26:14.731ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
6282025-04-30T06:26:14.731ZWARNtest_disk_expunge (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config nor servers
6292025-04-30T06:26:14.731ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 23.617\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6302025-04-30T06:26:14.731ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
6312025-04-30T06:26:14.731ZINFOtest_disk_expunge (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-04-30 06:16:14.731101492 UTC
6322025-04-30T06:26:14.731ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database
6332025-04-30T06:26:14.731ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
6342025-04-30T06:26:14.732ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
6352025-04-30T06:26:14.732ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
6362025-04-30T06:26:14.732ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
6372025-04-30T06:26:14.732ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6382025-04-30T06:26:14.732ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
6392025-04-30T06:26:14.732ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6402025-04-30T06:26:14.733ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6412025-04-30T06:26:14.733ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
6422025-04-30T06:26:14.733ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
6432025-04-30T06:26:14.733ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6442025-04-30T06:26:14.733ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6452025-04-30T06:26:14.734ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
6462025-04-30T06:26:14.734ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
6472025-04-30T06:26:14.734ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6482025-04-30T06:26:14.735ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
6492025-04-30T06:26:14.735ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
6502025-04-30T06:26:14.735ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
6512025-04-30T06:26:14.735ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig
6522025-04-30T06:26:14.735ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6532025-04-30T06:26:14.735ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
6542025-04-30T06:26:14.735ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6552025-04-30T06:26:14.737ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6562025-04-30T06:26:14.737ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
6572025-04-30T06:26:14.737ZWARNtest_disk_expunge (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = no blueprint
6582025-04-30T06:26:14.737ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 61.843\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6592025-04-30T06:26:14.737ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
6602025-04-30T06:26:14.737ZWARNtest_disk_expunge (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector reason = no blueprint
6612025-04-30T06:26:14.737ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 14.889\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6622025-04-30T06:26:14.737ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
6632025-04-30T06:26:14.737ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
6642025-04-30T06:26:14.737ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6652025-04-30T06:26:14.737ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
6662025-04-30T06:26:14.737ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
6672025-04-30T06:26:14.737ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
6682025-04-30T06:26:14.737ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
6692025-04-30T06:26:14.737ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
6702025-04-30T06:26:14.737ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6712025-04-30T06:26:14.737ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
6722025-04-30T06:26:14.737ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6732025-04-30T06:26:14.737ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6742025-04-30T06:26:14.737ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
6752025-04-30T06:26:14.738ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 18.988\u{b5}s iteration = 1 status = {"error":"task disabled"}
6762025-04-30T06:26:14.738ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
6772025-04-30T06:26:14.738ZWARNtest_disk_expunge (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
6782025-04-30T06:26:14.738ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 23.236\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6792025-04-30T06:26:14.738ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
6802025-04-30T06:26:14.738ZINFOtest_disk_expunge (BackgroundTasks): Decommissioned Disk Cleaner disabled background_task = decommissioned_disk_cleaner
6812025-04-30T06:26:14.738ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 83.607\u{b5}s iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
6822025-04-30T06:26:14.738ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
6832025-04-30T06:26:14.738ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
6842025-04-30T06:26:14.738ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6852025-04-30T06:26:14.738ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
6862025-04-30T06:26:14.738ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6872025-04-30T06:26:14.738ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6882025-04-30T06:26:14.739ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
6892025-04-30T06:26:14.739ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
6902025-04-30T06:26:14.739ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6912025-04-30T06:26:14.739ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
6922025-04-30T06:26:14.739ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6932025-04-30T06:26:14.740ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6942025-04-30T06:26:14.740ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
6952025-04-30T06:26:14.740ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
6962025-04-30T06:26:14.740ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6972025-04-30T06:26:14.741ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6982025-04-30T06:26:14.741ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
6992025-04-30T06:26:14.741ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
7002025-04-30T06:26:14.741ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7012025-04-30T06:26:14.741ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7022025-04-30T06:26:14.741ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
7032025-04-30T06:26:14.741ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
7042025-04-30T06:26:14.742ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7052025-04-30T06:26:14.742ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7062025-04-30T06:26:14.742ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
7072025-04-30T06:26:14.742ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
7082025-04-30T06:26:14.742ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7092025-04-30T06:26:14.742ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
7102025-04-30T06:26:14.742ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7112025-04-30T06:26:14.743ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7122025-04-30T06:26:14.743ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
7132025-04-30T06:26:14.743ZINFOtest_disk_expunge (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
7142025-04-30T06:26:14.743ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = instance_updater elapsed = 290.579\u{b5}s iteration = 1 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7152025-04-30T06:26:14.743ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
7162025-04-30T06:26:14.743ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
7172025-04-30T06:26:14.743ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7182025-04-30T06:26:14.744ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7192025-04-30T06:26:14.744ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
7202025-04-30T06:26:14.744ZDEBGtest_disk_expunge (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7212025-04-30T06:26:14.744ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
7222025-04-30T06:26:14.744ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7232025-04-30T06:26:14.744ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7242025-04-30T06:26:14.745ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
7252025-04-30T06:26:14.745ZINFOtest_disk_expunge (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
7262025-04-30T06:26:14.745ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
7272025-04-30T06:26:14.745ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7282025-04-30T06:26:14.745ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
7292025-04-30T06:26:14.745ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7302025-04-30T06:26:14.745ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7312025-04-30T06:26:14.745ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
7322025-04-30T06:26:14.745ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database
7332025-04-30T06:26:14.745ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
7342025-04-30T06:26:14.746ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
7352025-04-30T06:26:14.746ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
7362025-04-30T06:26:14.746ZDEBGtest_disk_expunge (SagaRecoverer): listing candidate sagas for recovery
7372025-04-30T06:26:14.746ZTRACtest_disk_expunge (SagaRecoverer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database
7382025-04-30T06:26:14.746ZDEBGtest_disk_expunge (SagaRecoverer): roles roles = RoleSet { roles: {} }
7392025-04-30T06:26:14.746ZDEBGtest_disk_expunge (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
7402025-04-30T06:26:14.746ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
7412025-04-30T06:26:14.746ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database
7422025-04-30T06:26:14.746ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
7432025-04-30T06:26:14.747ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
7442025-04-30T06:26:14.747ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
7452025-04-30T06:26:14.747ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
7462025-04-30T06:26:14.747ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
7472025-04-30T06:26:14.747ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
7482025-04-30T06:26:14.747ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
7492025-04-30T06:26:14.748ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database
7502025-04-30T06:26:14.748ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
7512025-04-30T06:26:14.748ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
7522025-04-30T06:26:14.748ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
7532025-04-30T06:26:14.748ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
7542025-04-30T06:26:14.748ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7552025-04-30T06:26:14.749ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
7562025-04-30T06:26:14.749ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
7572025-04-30T06:26:14.749ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database
7582025-04-30T06:26:14.749ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
7592025-04-30T06:26:14.749ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
7602025-04-30T06:26:14.749ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
7612025-04-30T06:26:14.750ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
7622025-04-30T06:26:14.750ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7632025-04-30T06:26:14.750ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
7642025-04-30T06:26:14.750ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
7652025-04-30T06:26:14.750ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
7662025-04-30T06:26:14.750ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
7672025-04-30T06:26:14.750ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database
7682025-04-30T06:26:14.750ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
7692025-04-30T06:26:14.751ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
7702025-04-30T06:26:14.751ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 1 reason = Timeout
7712025-04-30T06:26:14.751ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
7722025-04-30T06:26:14.751ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
7732025-04-30T06:26:14.751ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
7742025-04-30T06:26:14.751ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = webhook_deliverator iteration = 1 reason = Timeout
7752025-04-30T06:26:14.751ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database
7762025-04-30T06:26:14.752ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
7772025-04-30T06:26:14.752ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
7782025-04-30T06:26:14.752ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
7792025-04-30T06:26:14.752ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7802025-04-30T06:26:14.752ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
7812025-04-30T06:26:14.752ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7822025-04-30T06:26:14.753ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7832025-04-30T06:26:14.754ZINFOtest_disk_expunge (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpOa9xW8/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7842025-04-30T06:26:14.754ZDEBGtest_disk_expunge (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpOa9xW8/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7852025-04-30T06:26:14.754ZINFOtest_disk_expunge (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmp7u5XMP/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7862025-04-30T06:26:14.754ZDEBGtest_disk_expunge (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmp7u5XMP/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7872025-04-30T06:26:14.812ZDEBGtest_disk_expunge (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:47123 method = GET path = /artifact/sha256/{sha256} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7882025-04-30T06:26:14.812ZINFOtest_disk_expunge (dropshot (Repo Depot)): listening local_addr = [::1]:47123 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7892025-04-30T06:26:14.813ZDEBGtest_disk_expunge (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:47123 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7902025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /artifacts sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7912025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /artifacts/{sha256} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7922025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7932025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7942025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7952025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7962025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7972025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7982025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /bootstore/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7992025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8002025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /disks/{disk_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8012025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = POST path = /disks/{disk_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8022025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /eip-gateways sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8032025-04-30T06:26:14.822ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8042025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8052025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8062025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8072025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /omicron-physical-disks sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8082025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /sled-identifiers sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8092025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /sled-role sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8102025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /sleds sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8112025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support/dladm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8122025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support/ipadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8132025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support/logs/download/{zone} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8142025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support/logs/zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8152025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support/nvmeadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8162025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support/pargs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8172025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support/pfiles-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8182025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support/pstack-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8192025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support/zfs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8202025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support/zoneadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8212025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support/zpool-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8222025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8232025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8242025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8252025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8262025-04-30T06:26:14.823ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8272025-04-30T06:26:14.824ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8282025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8292025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8302025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8312025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8322025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /timesync sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8332025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = DELETE path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8342025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8352025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8362025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = DELETE path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8372025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8382025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8392025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8402025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8412025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = POST path = /vmms/{propolis_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8422025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8432025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8442025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8452025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8462025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8472025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8482025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8492025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8502025-04-30T06:26:14.825ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = POST path = /zones/bundle-cleanup sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8512025-04-30T06:26:14.826ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8522025-04-30T06:26:14.826ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = PUT path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8532025-04-30T06:26:14.826ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /zones/bundle-cleanup/utilization sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8542025-04-30T06:26:14.826ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /zones/bundles sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8552025-04-30T06:26:14.826ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /zones/bundles/{zone_name} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8562025-04-30T06:26:14.826ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8572025-04-30T06:26:14.826ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8582025-04-30T06:26:14.826ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:44559 method = GET path = /zpools sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8592025-04-30T06:26:14.826ZINFOtest_disk_expunge (dropshot): listening local_addr = [::1]:44559 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8602025-04-30T06:26:14.826ZDEBGtest_disk_expunge (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:44559 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8612025-04-30T06:26:14.826ZDEBGtest_disk_expunge (omicron_sled_agent::sim::Server): contacting server nexus sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8622025-04-30T06:26:14.827ZDEBGtest_disk_expunge (NexusClient): client request body = Some(Body) method = POST sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = http://127.0.0.1:34036/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8632025-04-30T06:26:14.828ZTRACtest_disk_expunge (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 28 c6 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 (............_na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:54959 req_id = 301721ae-f594-4e7c-90da-0af8a550226d
8642025-04-30T06:26:14.828ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10438,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54959 req_id = 301721ae-f594-4e7c-90da-0af8a550226d
8652025-04-30T06:26:14.828ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10438, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_nameservice._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 12, 95, 110, 97, 109, 101, 115, 101, 114, 118, 105, 99, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_nameservice._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54959 req_id = 301721ae-f594-4e7c-90da-0af8a550226d
8662025-04-30T06:26:14.829ZTRACtest_disk_expunge (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: c1 de 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61865 req_id = c6901773-3034-48c1-bb6b-a2ac4df300e7
8672025-04-30T06:26:14.829ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49630,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61865 req_id = c6901773-3034-48c1-bb6b-a2ac4df300e7
8682025-04-30T06:26:14.829ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49630, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_external-dns._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 13, 95, 101, 120, 116, 101, 114, 110, 97, 108, 45, 100, 110, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_external-dns._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61865 req_id = c6901773-3034-48c1-bb6b-a2ac4df300e7
8692025-04-30T06:26:14.829ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
8702025-04-30T06:26:14.829ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8712025-04-30T06:26:14.829ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
8722025-04-30T06:26:14.829ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8732025-04-30T06:26:14.830ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8742025-04-30T06:26:14.830ZINFOtest_disk_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:34036 remote_addr = 127.0.0.1:40665
8752025-04-30T06:26:14.831ZWARNtest_disk_expunge (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_internal dns_group = internal error = no record found for Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8762025-04-30T06:26:14.831ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 101.658117ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_nameservice._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8772025-04-30T06:26:14.831ZWARNtest_disk_expunge (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_external dns_group = external error = no record found for Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8782025-04-30T06:26:14.831ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 100.854325ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_external-dns._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8792025-04-30T06:26:14.839ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:40665 req_id = 441f4edc-5021-4de7-b2a4-8fd076895b50 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8802025-04-30T06:26:14.842ZINFOtest_disk_expunge (nexus): registered sled agent sled_uuid = b6d65341-167c-41df-9b5c-41cded99c229
8812025-04-30T06:26:14.844ZINFOtest_disk_expunge (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-04-30 06:16:14.731101492 UTC pruned = {}
8822025-04-30T06:26:14.844ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 113.687515ms iteration = 1 status = {"expiration":"2025-04-30T06:16:14.731101492Z","pruned":[]}
8832025-04-30T06:26:14.848ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8842025-04-30T06:26:14.852ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8852025-04-30T06:26:14.859ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8862025-04-30T06:26:14.863ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
8872025-04-30T06:26:14.863ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8882025-04-30T06:26:14.864ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8892025-04-30T06:26:14.864ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8902025-04-30T06:26:14.876ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 141.3797ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
8912025-04-30T06:26:14.885ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 46 63 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 Fc..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:60592 req_id = 50ed67fd-bf41-4ded-961b-9ca3529bfa95
8922025-04-30T06:26:14.885ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18019,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60592 req_id = 50ed67fd-bf41-4ded-961b-9ca3529bfa95
8932025-04-30T06:26:14.885ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18019, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60592 req_id = 50ed67fd-bf41-4ded-961b-9ca3529bfa95
8942025-04-30T06:26:14.920ZERROtest_disk_expunge (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8952025-04-30T06:26:14.920ZERROtest_disk_expunge (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8962025-04-30T06:26:14.920ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 187.050807ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8972025-04-30T06:26:14.920ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: de 1c 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63203 req_id = 9f9ba0fe-9604-4520-807f-3e9db7396dd3
8982025-04-30T06:26:14.925ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56860,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63203 req_id = 9f9ba0fe-9604-4520-807f-3e9db7396dd3
8992025-04-30T06:26:14.925ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56860, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63203 req_id = 9f9ba0fe-9604-4520-807f-3e9db7396dd3
9002025-04-30T06:26:14.926ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9012025-04-30T06:26:14.930ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9022025-04-30T06:26:14.930ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9032025-04-30T06:26:14.930ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9042025-04-30T06:26:14.931ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9052025-04-30T06:26:14.931ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9062025-04-30T06:26:14.936ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
9072025-04-30T06:26:14.936ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
9082025-04-30T06:26:14.936ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9092025-04-30T06:26:14.937ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
9102025-04-30T06:26:14.937ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9112025-04-30T06:26:14.943ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9122025-04-30T06:26:14.945ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
9132025-04-30T06:26:14.946ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9142025-04-30T06:26:14.947ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9152025-04-30T06:26:14.947ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9162025-04-30T06:26:14.953ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9172025-04-30T06:26:14.960ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
9182025-04-30T06:26:14.960ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9192025-04-30T06:26:14.961ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9202025-04-30T06:26:14.962ZERROtest_disk_expunge (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9212025-04-30T06:26:14.962ZERROtest_disk_expunge (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9222025-04-30T06:26:14.962ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 228.026929ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9232025-04-30T06:26:15.142ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9242025-04-30T06:26:15.142ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9252025-04-30T06:26:15.148ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9262025-04-30T06:26:15.149ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9272025-04-30T06:26:15.158ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9282025-04-30T06:26:15.159ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
9292025-04-30T06:26:15.159ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9302025-04-30T06:26:15.167ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9312025-04-30T06:26:15.168ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9322025-04-30T06:26:15.172ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9332025-04-30T06:26:15.173ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
9342025-04-30T06:26:15.173ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9352025-04-30T06:26:15.173ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
9362025-04-30T06:26:15.173ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9372025-04-30T06:26:15.173ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9382025-04-30T06:26:15.174ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9392025-04-30T06:26:15.180ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9402025-04-30T06:26:15.180ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
9412025-04-30T06:26:15.180ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9422025-04-30T06:26:15.180ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9432025-04-30T06:26:15.199ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
9442025-04-30T06:26:15.199ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9452025-04-30T06:26:15.200ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9462025-04-30T06:26:15.290ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 544.00297ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
9472025-04-30T06:26:15.309ZDEBGtest_disk_expunge (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
9482025-04-30T06:26:15.310ZINFOtest_disk_expunge (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
9492025-04-30T06:26:15.310ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 564.494819ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
9502025-04-30T06:26:15.310ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9512025-04-30T06:26:15.314ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9522025-04-30T06:26:15.314ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
9532025-04-30T06:26:15.314ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9542025-04-30T06:26:15.314ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
9552025-04-30T06:26:15.314ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9562025-04-30T06:26:15.317ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9572025-04-30T06:26:15.381ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9582025-04-30T06:26:15.389ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9592025-04-30T06:26:15.389ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
9602025-04-30T06:26:15.389ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9612025-04-30T06:26:15.390ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9622025-04-30T06:26:15.426ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 674.746498ms iteration = 1 status = {"by_rx":{},"error":null}
9632025-04-30T06:26:15.539ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
9642025-04-30T06:26:15.539ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9652025-04-30T06:26:15.540ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9662025-04-30T06:26:15.551ZTRACtest_disk_expunge (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = instance failed total_found = 0
9672025-04-30T06:26:15.551ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
9682025-04-30T06:26:15.551ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
9692025-04-30T06:26:15.552ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9702025-04-30T06:26:15.561ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
9712025-04-30T06:26:15.561ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9722025-04-30T06:26:15.561ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9732025-04-30T06:26:15.561ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9742025-04-30T06:26:15.562ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9752025-04-30T06:26:15.566ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 827.004984ms iteration = 1 status = {}
9762025-04-30T06:26:15.601ZINFOtest_disk_expunge (SagaRecoverer): listed in-progress sagas count = 0
9772025-04-30T06:26:15.601ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 855.22991ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
9782025-04-30T06:26:15.602ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9792025-04-30T06:26:15.602ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9802025-04-30T06:26:15.603ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9812025-04-30T06:26:15.608ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
9822025-04-30T06:26:15.608ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
9832025-04-30T06:26:15.608ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 9b b4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54164 req_id = 20814e31-8a97-482e-8aff-7c6fd497cfa0
9842025-04-30T06:26:15.608ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39860,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54164 req_id = 20814e31-8a97-482e-8aff-7c6fd497cfa0
9852025-04-30T06:26:15.609ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39860, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54164 req_id = 20814e31-8a97-482e-8aff-7c6fd497cfa0
9862025-04-30T06:26:15.609ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9872025-04-30T06:26:15.609ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 873.396074ms iteration = 1 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9882025-04-30T06:26:15.610ZWARNtest_disk_expunge (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set
9892025-04-30T06:26:15.610ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 875.364997ms iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
9902025-04-30T06:26:15.612ZWARNtest_disk_expunge (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker
9912025-04-30T06:26:15.612ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 874.39217ms iteration = 1 status = {"error":"inventory collection is None"}
9922025-04-30T06:26:15.621ZWARNtest_disk_expunge (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0
9932025-04-30T06:26:15.621ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 892.634176ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
9942025-04-30T06:26:15.623ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
9952025-04-30T06:26:15.623ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9962025-04-30T06:26:15.623ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9972025-04-30T06:26:15.624ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9982025-04-30T06:26:15.629ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9992025-04-30T06:26:15.629ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 879.188366ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
10002025-04-30T06:26:15.630ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 880.765628ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
10012025-04-30T06:26:15.631ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 883.353424ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
10022025-04-30T06:26:15.645ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10032025-04-30T06:26:15.653ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10042025-04-30T06:26:15.653ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10052025-04-30T06:26:15.653ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10062025-04-30T06:26:15.655ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10072025-04-30T06:26:15.724ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 977.108384ms iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10082025-04-30T06:26:15.727ZTRACtest_disk_expunge (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = webhook_dispatcher
10092025-04-30T06:26:15.727ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
10102025-04-30T06:26:15.727ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
10112025-04-30T06:26:15.728ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
10122025-04-30T06:26:15.730ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10132025-04-30T06:26:15.736ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
10142025-04-30T06:26:15.736ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
10152025-04-30T06:26:15.736ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10162025-04-30T06:26:15.737ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10172025-04-30T06:26:15.768ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
10182025-04-30T06:26:15.768ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
10192025-04-30T06:26:15.768ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10202025-04-30T06:26:15.768ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10212025-04-30T06:26:15.769ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10222025-04-30T06:26:15.786ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
10232025-04-30T06:26:15.786ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10242025-04-30T06:26:15.787ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10252025-04-30T06:26:15.787ZWARNtest_disk_expunge (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0
10262025-04-30T06:26:15.787ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 1.056962728s iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
10272025-04-30T06:26:15.827ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
10282025-04-30T06:26:15.827ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10292025-04-30T06:26:15.828ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10302025-04-30T06:26:15.832ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10312025-04-30T06:26:15.842ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10322025-04-30T06:26:15.845ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
10332025-04-30T06:26:15.845ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10342025-04-30T06:26:15.846ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10352025-04-30T06:26:15.893ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Signaled
10362025-04-30T06:26:15.893ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
10372025-04-30T06:26:15.893ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10382025-04-30T06:26:15.893ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10392025-04-30T06:26:15.894ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10402025-04-30T06:26:15.894ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10412025-04-30T06:26:15.895ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 1055371 local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:40665 req_id = 441f4edc-5021-4de7-b2a4-8fd076895b50 response_code = 204 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
10422025-04-30T06:26:15.895ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
10432025-04-30T06:26:15.895ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10442025-04-30T06:26:15.895ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10452025-04-30T06:26:15.895ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10462025-04-30T06:26:15.896ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10472025-04-30T06:26:15.896ZINFOtest_disk_expunge (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10482025-04-30T06:26:15.896ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 1.154254215s iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10492025-04-30T06:26:15.896ZTRACtest_disk_expunge (BackgroundTasks): webhook dispatching completed successfully background_task = webhook_dispatcher events_dispatched = 0 events_without_receivers = 0
10502025-04-30T06:26:15.896ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 1.145705082s iteration = 1 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
10512025-04-30T06:26:15.898ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = region_replacement elapsed = 1.157595102s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10522025-04-30T06:26:15.898ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 1.157107079s iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10532025-04-30T06:26:15.899ZWARNtest_disk_expunge (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone
10542025-04-30T06:26:15.899ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 1.166790594s iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
10552025-04-30T06:26:15.899ZDEBGtest_disk_expunge (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:34036/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229", status: 204, headers: {"x-request-id": "441f4edc-5021-4de7-b2a4-8fd076895b50", "date": "Wed, 30 Apr 2025 06:26:15 GMT"} }) sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10562025-04-30T06:26:15.899ZDEBGtest_disk_expunge (init_with_steps): Running step start_crucible_pantry
10572025-04-30T06:26:15.901ZDEBGtest_disk_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:47737 method = GET path = /crucible/pantry/0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10582025-04-30T06:26:15.901ZDEBGtest_disk_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:47737 method = GET path = /crucible/pantry/0/job/{id}/is-finished sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10592025-04-30T06:26:15.901ZDEBGtest_disk_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:47737 method = GET path = /crucible/pantry/0/job/{id}/ok sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10602025-04-30T06:26:15.901ZDEBGtest_disk_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:47737 method = DELETE path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10612025-04-30T06:26:15.901ZDEBGtest_disk_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:47737 method = GET path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10622025-04-30T06:26:15.901ZDEBGtest_disk_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:47737 method = POST path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10632025-04-30T06:26:15.901ZDEBGtest_disk_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:47737 method = POST path = /crucible/pantry/0/volume/{id}/background sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10642025-04-30T06:26:15.901ZDEBGtest_disk_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:47737 method = POST path = /crucible/pantry/0/volume/{id}/bulk-write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10652025-04-30T06:26:15.901ZDEBGtest_disk_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:47737 method = POST path = /crucible/pantry/0/volume/{id}/import-from-url sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10662025-04-30T06:26:15.901ZDEBGtest_disk_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:47737 method = POST path = /crucible/pantry/0/volume/{id}/scrub sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10672025-04-30T06:26:15.901ZDEBGtest_disk_expunge (dropshot): registered endpoint kind = pantry local_addr = [::1]:47737 method = POST path = /crucible/pantry/0/volume/{id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10682025-04-30T06:26:15.901ZINFOtest_disk_expunge (dropshot): listening kind = pantry local_addr = [::1]:47737 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10692025-04-30T06:26:15.902ZDEBGtest_disk_expunge (dropshot): successfully registered DTrace USDT probes kind = pantry local_addr = [::1]:47737 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10702025-04-30T06:26:15.902ZINFOtest_disk_expunge (omicron_sled_agent::sim::Server): Started Simulated Crucible Pantry address = [::1]:47737 kind = pantry sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10712025-04-30T06:26:15.902ZDEBGtest_disk_expunge (init_with_steps): Running step populate_internal_dns
10722025-04-30T06:26:15.902ZDEBGtest_disk_expunge: Populating Internal DNS
10732025-04-30T06:26:15.962ZINFOtest_disk_expunge: DNS population: DnsConfigParams { generation: Generation( 1, ), time_created: 2025-04-30T06:26:15.961573336Z, zones: [ DnsConfigZone { zone_name: "control-plane.oxide.internal", records: { "78121327-166d-4e33-935c-82ab8635bf33.host": [ Aaaa( ::ffff:127.0.0.1, ), ], "_nexus._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 34036, target: "78121327-166d-4e33-935c-82ab8635bf33.host.control-plane.oxide.internal", }, ), ], "b6d65341-167c-41df-9b5c-41cded99c229.sled": [ Aaaa( ::1, ), ], "116bed1f-7c51-4865-8a2e-dc625a6fb25d.host": [ Aaaa( ::1, ), ], "_nameservice._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 36493, target: "2784c366-c956-4ffa-afcb-17669db9c967.host.control-plane.oxide.internal", }, ), ], "_clickhouse-native._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "61098437-8bb8-470a-9c29-11487743fb5a.host.control-plane.oxide.internal", }, ), ], "_crucible-pantry._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 47737, target: "116bed1f-7c51-4865-8a2e-dc625a6fb25d.host.control-plane.oxide.internal", }, ), ], "_repo-depot._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 47123, target: "b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal", }, ), ], "_external-dns._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 52024, target: "68c89553-4ad9-4464-83e9-6636df41c29b.host.control-plane.oxide.internal", }, ), ], "2784c366-c956-4ffa-afcb-17669db9c967.host": [ Aaaa( ::1, ), ], "382008af-ff99-4ede-97fe-7a091b9b6c59.host": [ Aaaa( ::1, ), ], "_clickhouse._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 41109, target: "61098437-8bb8-470a-9c29-11487743fb5a.host.control-plane.oxide.internal", }, ), ], "_clickhouse-admin-single-server._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 8888, target: "61098437-8bb8-470a-9c29-11487743fb5a.host.control-plane.oxide.internal", }, ), ], "68c89553-4ad9-4464-83e9-6636df41c29b.host": [ Aaaa( ::1, ), ], "_cockroach._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 63866, target: "382008af-ff99-4ede-97fe-7a091b9b6c59.host.control-plane.oxide.internal", }, ), ], "_oximeter-reader._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "61098437-8bb8-470a-9c29-11487743fb5a.host.control-plane.oxide.internal", }, ), ], "_mgd._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 61919, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_mgs._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 41990, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host": [ Aaaa( ::1, ), ], "61098437-8bb8-470a-9c29-11487743fb5a.host": [ Aaaa( ::1, ), ], "_dendrite._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 55614, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], }, }, ], }
10742025-04-30T06:26:15.963ZDEBGtest_disk_expunge: client request body = Some(Body) method = PUT uri = http://[::1]:36493/config
10752025-04-30T06:26:15.966ZINFOtest_disk_expunge (http): accepted connection kind = dns local_addr = [::1]:36493 remote_addr = [::1]:54168
10762025-04-30T06:26:15.968ZTRACtest_disk_expunge (http): incoming request kind = dns local_addr = [::1]:36493 method = PUT remote_addr = [::1]:54168 req_id = 41fc36e0-9677-4295-ac17-ea9cc1b6e79c uri = /config
10772025-04-30T06:26:15.969ZINFOtest_disk_expunge (store): attempting generation update new_generation = 1 req_id = 41fc36e0-9677-4295-ac17-ea9cc1b6e79c
10782025-04-30T06:26:15.969ZINFOtest_disk_expunge (store): pruning trees for generations newer than 0
10792025-04-30T06:26:15.969ZDEBGtest_disk_expunge (store): creating tree tree_name = generation_1_zone_control-plane.oxide.internal
10802025-04-30T06:26:15.970ZDEBGtest_disk_expunge (store): updating current config
10812025-04-30T06:26:15.970ZDEBGtest_disk_expunge (store): flushing default tree
10822025-04-30T06:26:15.970ZINFOtest_disk_expunge (store): pruning trees for generations older than 1 keep = 3
10832025-04-30T06:26:15.970ZINFOtest_disk_expunge (store): updated generation new_generation = 1 req_id = 41fc36e0-9677-4295-ac17-ea9cc1b6e79c
10842025-04-30T06:26:15.970ZINFOtest_disk_expunge (http): request completed kind = dns latency_us = 2507 local_addr = [::1]:36493 method = PUT remote_addr = [::1]:54168 req_id = 41fc36e0-9677-4295-ac17-ea9cc1b6e79c response_code = 204 uri = /config
10852025-04-30T06:26:15.971ZDEBGtest_disk_expunge: client response result = Ok(Response { url: "http://[::1]:36493/config", status: 204, headers: {"x-request-id": "41fc36e0-9677-4295-ac17-ea9cc1b6e79c", "date": "Wed, 30 Apr 2025 06:26:15 GMT"} })
10862025-04-30T06:26:15.972ZDEBGtest_disk_expunge (init_with_steps): Running step configure_sled_agents
10872025-04-30T06:26:16.041ZDEBGtest_disk_expunge: client request body = Some(Body) method = PUT uri = http://[::1]:44559/omicron-config
10882025-04-30T06:26:16.097ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
10892025-04-30T06:26:16.097ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10902025-04-30T06:26:16.097ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10912025-04-30T06:26:16.097ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10922025-04-30T06:26:16.098ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10932025-04-30T06:26:16.098ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
10942025-04-30T06:26:16.098ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10952025-04-30T06:26:16.098ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10962025-04-30T06:26:16.098ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10972025-04-30T06:26:16.099ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 1.350352248s iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
10982025-04-30T06:26:16.099ZTRACtest_disk_expunge (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = start saga failed total_found = 0
10992025-04-30T06:26:16.099ZINFOtest_disk_expunge (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
11002025-04-30T06:26:16.099ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 1.356016523s iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11012025-04-30T06:26:16.099ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:64803 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11022025-04-30T06:26:16.103ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = PUT remote_addr = [::1]:64803 req_id = a27a4553-9957-4b1b-a767-ce047028b059 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
11032025-04-30T06:26:16.117ZINFOtest_disk_expunge (dropshot): request completed latency_us = 14438 local_addr = [::1]:44559 method = PUT remote_addr = [::1]:64803 req_id = a27a4553-9957-4b1b-a767-ce047028b059 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
11042025-04-30T06:26:16.132ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11052025-04-30T06:26:16.139ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
11062025-04-30T06:26:16.146ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
11072025-04-30T06:26:16.146ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
11082025-04-30T06:26:16.148ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
11092025-04-30T06:26:16.149ZDEBGtest_disk_expunge: client response result = Ok(Response { url: "http://[::1]:44559/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "a27a4553-9957-4b1b-a767-ce047028b059", "content-length": "26", "date": "Wed, 30 Apr 2025 06:26:15 GMT"} })
11102025-04-30T06:26:16.150ZDEBGtest_disk_expunge: client request body = Some(Body) method = PUT uri = http://[::1]:44559/network-bootstore-config
11112025-04-30T06:26:16.173ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = PUT remote_addr = [::1]:64803 req_id = e6d5fb74-8ef7-4085-9af3-da40d1b8efb7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
11122025-04-30T06:26:16.174ZINFOtest_disk_expunge (dropshot): request completed latency_us = 919 local_addr = [::1]:44559 method = PUT remote_addr = [::1]:64803 req_id = e6d5fb74-8ef7-4085-9af3-da40d1b8efb7 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
11132025-04-30T06:26:16.177ZDEBGtest_disk_expunge: client response result = Ok(Response { url: "http://[::1]:44559/network-bootstore-config", status: 204, headers: {"x-request-id": "e6d5fb74-8ef7-4085-9af3-da40d1b8efb7", "date": "Wed, 30 Apr 2025 06:26:15 GMT"} })
11142025-04-30T06:26:16.178ZDEBGtest_disk_expunge (init_with_steps): Running step start_nexus_external
11152025-04-30T06:26:16.178ZDEBGtest_disk_expunge: Starting Nexus (external API)
11162025-04-30T06:26:19.071ZTRACtest_disk_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
11172025-04-30T06:26:19.072ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11182025-04-30T06:26:19.072ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11192025-04-30T06:26:19.072ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
11202025-04-30T06:26:19.073ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11212025-04-30T06:26:19.075ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11222025-04-30T06:26:19.082ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11232025-04-30T06:26:19.082ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
11242025-04-30T06:26:19.082ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11252025-04-30T06:26:19.082ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11262025-04-30T06:26:19.084ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11272025-04-30T06:26:19.085ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11282025-04-30T06:26:19.085ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11292025-04-30T06:26:19.085ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11302025-04-30T06:26:19.086ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11312025-04-30T06:26:19.171ZINFOtest_disk_expunge (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
11322025-04-30T06:26:19.171ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:44559/vpc-routes
11332025-04-30T06:26:19.172ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11342025-04-30T06:26:19.177ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11352025-04-30T06:26:19.177ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11362025-04-30T06:26:19.177ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11372025-04-30T06:26:19.178ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11382025-04-30T06:26:19.179ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11392025-04-30T06:26:19.179ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11402025-04-30T06:26:19.179ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11412025-04-30T06:26:19.179ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11422025-04-30T06:26:19.179ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
11432025-04-30T06:26:19.179ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
11442025-04-30T06:26:19.181ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:46156 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11452025-04-30T06:26:19.192ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:46156 req_id = 5bb66946-932d-4c21-9733-23ba15e4c041 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11462025-04-30T06:26:19.192ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d4 92 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52426 req_id = 7e217239-acaf-49fa-8881-74097e726a38
11472025-04-30T06:26:19.192ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54418,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52426 req_id = 7e217239-acaf-49fa-8881-74097e726a38
11482025-04-30T06:26:19.193ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
11492025-04-30T06:26:19.209ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11502025-04-30T06:26:19.209ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52426 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7e217239-acaf-49fa-8881-74097e726a38
11512025-04-30T06:26:19.210ZINFOtest_disk_expunge (dropshot): request completed latency_us = 18293 local_addr = [::1]:44559 method = GET remote_addr = [::1]:46156 req_id = 5bb66946-932d-4c21-9733-23ba15e4c041 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11522025-04-30T06:26:19.211ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11532025-04-30T06:26:19.215ZDEBGtest_disk_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
11542025-04-30T06:26:19.216ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11552025-04-30T06:26:19.216ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
11562025-04-30T06:26:19.216ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11572025-04-30T06:26:19.218ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3244, tv_nsec: 17589544 } })
11582025-04-30T06:26:19.219ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
11592025-04-30T06:26:19.221ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:44559/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bb66946-932d-4c21-9733-23ba15e4c041", "content-length": "2", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
11602025-04-30T06:26:19.221ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
11612025-04-30T06:26:19.221ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11622025-04-30T06:26:19.222ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11632025-04-30T06:26:19.228ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11642025-04-30T06:26:19.233ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11652025-04-30T06:26:19.233ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11662025-04-30T06:26:19.233ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11672025-04-30T06:26:19.233ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11682025-04-30T06:26:19.233ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11692025-04-30T06:26:19.233ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11702025-04-30T06:26:19.233ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11712025-04-30T06:26:19.233ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11722025-04-30T06:26:19.233ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11732025-04-30T06:26:19.233ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11742025-04-30T06:26:19.234ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11752025-04-30T06:26:19.235ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11762025-04-30T06:26:19.242ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
11772025-04-30T06:26:19.242ZTRACtest_disk_expunge (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
11782025-04-30T06:26:19.242ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
11792025-04-30T06:26:19.242ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
11802025-04-30T06:26:19.242ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11812025-04-30T06:26:19.242ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11822025-04-30T06:26:19.242ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11832025-04-30T06:26:19.262ZINFOtest_disk_expunge (BackgroundTasks): No bundles to collect background_task = support_bundle_collector
11842025-04-30T06:26:19.262ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 4.525088513s iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
11852025-04-30T06:26:19.276ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 3b 06 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ;...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51241 req_id = 957a8dce-7187-475e-b130-c0fe9b9c96af
11862025-04-30T06:26:19.276ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15110,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51241 req_id = 957a8dce-7187-475e-b130-c0fe9b9c96af
11872025-04-30T06:26:19.276ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11882025-04-30T06:26:19.276ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51241 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 957a8dce-7187-475e-b130-c0fe9b9c96af
11892025-04-30T06:26:19.340ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
11902025-04-30T06:26:19.341ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11912025-04-30T06:26:19.341ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
11922025-04-30T06:26:19.352ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11932025-04-30T06:26:19.359ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 61 e8 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c a............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52290 req_id = 328c4170-f128-40c1-8cd4-52495cc94883
11942025-04-30T06:26:19.359ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25064,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52290 req_id = 328c4170-f128-40c1-8cd4-52495cc94883
11952025-04-30T06:26:19.359ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
11962025-04-30T06:26:19.359ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25064, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52290 req_id = 328c4170-f128-40c1-8cd4-52495cc94883
11972025-04-30T06:26:19.360ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11982025-04-30T06:26:19.364ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11992025-04-30T06:26:19.364ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
12002025-04-30T06:26:19.364ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12012025-04-30T06:26:19.364ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12022025-04-30T06:26:19.365ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12032025-04-30T06:26:19.369ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
12042025-04-30T06:26:19.369ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12052025-04-30T06:26:19.369ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12062025-04-30T06:26:19.370ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12072025-04-30T06:26:19.371ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
12082025-04-30T06:26:19.371ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
12092025-04-30T06:26:19.371ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
12102025-04-30T06:26:19.377ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12112025-04-30T06:26:19.377ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12122025-04-30T06:26:19.377ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12132025-04-30T06:26:19.378ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12142025-04-30T06:26:19.449ZINFOtest_disk_expunge (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
12152025-04-30T06:26:19.450ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:44559/eip-gateways
12162025-04-30T06:26:19.488ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = PUT remote_addr = [::1]:46156 req_id = ea80420b-96fb-49cc-9f75-903c1aa7eb80 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12172025-04-30T06:26:19.488ZINFOtest_disk_expunge (dropshot): request completed latency_us = 659 local_addr = [::1]:44559 method = PUT remote_addr = [::1]:46156 req_id = ea80420b-96fb-49cc-9f75-903c1aa7eb80 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12182025-04-30T06:26:19.489ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[::1]:44559/artifacts-config
12192025-04-30T06:26:19.489ZINFOtest_disk_expunge (InternalApi): set cockroachdb setting setting = cluster.preserve_downgrade_option value = 22.1
12202025-04-30T06:26:19.489ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12212025-04-30T06:26:19.489ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
12222025-04-30T06:26:19.490ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12232025-04-30T06:26:19.491ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:44675 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12242025-04-30T06:26:19.495ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:44559/eip-gateways", status: 204, headers: {"x-request-id": "ea80420b-96fb-49cc-9f75-903c1aa7eb80", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
12252025-04-30T06:26:19.496ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 4.751430966s iteration = 1 status = {}
12262025-04-30T06:26:19.497ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12272025-04-30T06:26:19.497ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12282025-04-30T06:26:19.498ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12292025-04-30T06:26:19.498ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12302025-04-30T06:26:19.514ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = PUT remote_addr = [::1]:44675 req_id = 8f8974e7-b98c-408e-94ab-c5410d996e23 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12312025-04-30T06:26:19.514ZINFOtest_disk_expunge (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpOa9xW8/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12322025-04-30T06:26:19.514ZDEBGtest_disk_expunge (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpOa9xW8/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12332025-04-30T06:26:19.514ZINFOtest_disk_expunge (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmp7u5XMP/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12342025-04-30T06:26:19.514ZDEBGtest_disk_expunge (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmp7u5XMP/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12352025-04-30T06:26:19.514ZINFOtest_disk_expunge (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpOa9xW8/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12362025-04-30T06:26:19.526ZINFOtest_disk_expunge (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmp7u5XMP/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12372025-04-30T06:26:19.540ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12382025-04-30T06:26:19.565ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12392025-04-30T06:26:19.565ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12402025-04-30T06:26:19.565ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12412025-04-30T06:26:19.566ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12422025-04-30T06:26:19.638ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:44559/v2p
12432025-04-30T06:26:19.640ZINFOtest_disk_expunge (ArtifactStore): Starting delete reconciler generation = 1 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12442025-04-30T06:26:19.640ZINFOtest_disk_expunge (dropshot): request completed latency_us = 126799 local_addr = [::1]:44559 method = PUT remote_addr = [::1]:44675 req_id = 8f8974e7-b98c-408e-94ab-c5410d996e23 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12452025-04-30T06:26:19.641ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
12462025-04-30T06:26:19.641ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12472025-04-30T06:26:19.641ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12482025-04-30T06:26:19.641ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
12492025-04-30T06:26:19.642ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12502025-04-30T06:26:19.643ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:46118 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12512025-04-30T06:26:19.643ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:44559/artifacts-config", status: 204, headers: {"x-request-id": "8f8974e7-b98c-408e-94ab-c5410d996e23", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
12522025-04-30T06:26:19.643ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = None method = GET uri = http://[::1]:44559/artifacts
12532025-04-30T06:26:19.645ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12542025-04-30T06:26:19.645ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12552025-04-30T06:26:19.646ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12562025-04-30T06:26:19.663ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:44675 req_id = bb76f9ea-b0e7-4139-8e69-dad081bfefdb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12572025-04-30T06:26:19.664ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:46118 req_id = 6bc1ab06-bb0e-47c9-a59a-ee6cc1aa8995 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12582025-04-30T06:26:19.665ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1959 local_addr = [::1]:44559 method = GET remote_addr = [::1]:44675 req_id = bb76f9ea-b0e7-4139-8e69-dad081bfefdb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12592025-04-30T06:26:19.665ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1761 local_addr = [::1]:44559 method = GET remote_addr = [::1]:46118 req_id = 6bc1ab06-bb0e-47c9-a59a-ee6cc1aa8995 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12602025-04-30T06:26:19.665ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12612025-04-30T06:26:19.666ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12622025-04-30T06:26:19.666ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12632025-04-30T06:26:19.667ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12642025-04-30T06:26:19.672ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
12652025-04-30T06:26:19.673ZDEBGtest_disk_expunge (InternalApi): updating rack subnet for rack c19a698f-c6f9-4a17-ae30-20d711b8f7dc to Some( V6( Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, ), )
12662025-04-30T06:26:19.673ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12672025-04-30T06:26:19.673ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
12682025-04-30T06:26:19.673ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12692025-04-30T06:26:19.706ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "657cd1f7-f729-4af2-91a6-5eb1ce7dc484", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
12702025-04-30T06:26:19.707ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
12712025-04-30T06:26:19.708ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:44559/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bc1ab06-bb0e-47c9-a59a-ee6cc1aa8995", "content-length": "2", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
12722025-04-30T06:26:19.708ZINFOtest_disk_expunge (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
12732025-04-30T06:26:19.708ZINFOtest_disk_expunge (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
12742025-04-30T06:26:19.708ZINFOtest_disk_expunge (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
12752025-04-30T06:26:19.709ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 4.969715904s iteration = 1 status = {}
12762025-04-30T06:26:19.710ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:44559/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb76f9ea-b0e7-4139-8e69-dad081bfefdb", "content-length": "26", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
12772025-04-30T06:26:19.710ZINFOtest_disk_expunge (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication sled = http://[::1]:44559
12782025-04-30T06:26:19.713ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 4.963136936s iteration = 1 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:26:19.643767417Z"},{"operation":"list","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:26:19.710157024Z"}]}
12792025-04-30T06:26:19.713ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12802025-04-30T06:26:19.713ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12812025-04-30T06:26:19.714ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12822025-04-30T06:26:19.729ZINFOtest_disk_expunge (InternalApi): Using static configuration for external switchports
12832025-04-30T06:26:19.730ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12842025-04-30T06:26:19.730ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
12852025-04-30T06:26:19.732ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12862025-04-30T06:26:19.733ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12872025-04-30T06:26:19.733ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12882025-04-30T06:26:19.733ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12892025-04-30T06:26:19.733ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12902025-04-30T06:26:19.734ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12912025-04-30T06:26:19.746ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12922025-04-30T06:26:19.746ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12932025-04-30T06:26:19.746ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12942025-04-30T06:26:19.747ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12952025-04-30T06:26:19.769ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12962025-04-30T06:26:19.769ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12972025-04-30T06:26:19.769ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12982025-04-30T06:26:19.770ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12992025-04-30T06:26:19.774ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13002025-04-30T06:26:19.780ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
13012025-04-30T06:26:19.781ZTRACtest_disk_expunge (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
13022025-04-30T06:26:19.781ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13032025-04-30T06:26:19.781ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13042025-04-30T06:26:19.781ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13052025-04-30T06:26:19.782ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58bee9ab-0850-447b-9665-ff0ff9c99f8d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
13062025-04-30T06:26:19.783ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
13072025-04-30T06:26:19.787ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13082025-04-30T06:26:19.787ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13092025-04-30T06:26:19.787ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13102025-04-30T06:26:19.803ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13112025-04-30T06:26:19.803ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13122025-04-30T06:26:19.805ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13132025-04-30T06:26:19.813ZINFOtest_disk_expunge (InternalApi): Recording Rack Network Configuration
13142025-04-30T06:26:19.813ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13152025-04-30T06:26:19.813ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
13162025-04-30T06:26:19.814ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13172025-04-30T06:26:19.824ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13182025-04-30T06:26:19.824ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13192025-04-30T06:26:19.825ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13202025-04-30T06:26:19.834ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13212025-04-30T06:26:19.834ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13222025-04-30T06:26:19.834ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13232025-04-30T06:26:19.834ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13242025-04-30T06:26:19.835ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13252025-04-30T06:26:19.840ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13262025-04-30T06:26:19.840ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13272025-04-30T06:26:19.840ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13282025-04-30T06:26:19.841ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13292025-04-30T06:26:19.852ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02241789-3d59-470f-991a-bdef011c31a5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
13302025-04-30T06:26:19.852ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
13312025-04-30T06:26:19.865ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13322025-04-30T06:26:19.865ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13332025-04-30T06:26:19.865ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13342025-04-30T06:26:19.866ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13352025-04-30T06:26:19.869ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13362025-04-30T06:26:19.877ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13372025-04-30T06:26:19.879ZTRACtest_disk_expunge (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
13382025-04-30T06:26:19.880ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13392025-04-30T06:26:19.880ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13402025-04-30T06:26:19.880ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13412025-04-30T06:26:19.880ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13422025-04-30T06:26:19.881ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13432025-04-30T06:26:19.887ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13442025-04-30T06:26:19.887ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13452025-04-30T06:26:19.887ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13462025-04-30T06:26:19.887ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13472025-04-30T06:26:19.898ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13482025-04-30T06:26:19.898ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13492025-04-30T06:26:19.898ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13502025-04-30T06:26:19.899ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13512025-04-30T06:26:19.900ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdac9e86-4f3a-47e2-b9b6-57860d2829d5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
13522025-04-30T06:26:19.900ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
13532025-04-30T06:26:19.903ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13542025-04-30T06:26:19.915ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13552025-04-30T06:26:19.915ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13562025-04-30T06:26:19.915ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13572025-04-30T06:26:19.917ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13582025-04-30T06:26:19.934ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13592025-04-30T06:26:19.934ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13602025-04-30T06:26:19.935ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13612025-04-30T06:26:19.936ZTRACtest_disk_expunge (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
13622025-04-30T06:26:19.936ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13632025-04-30T06:26:19.936ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13642025-04-30T06:26:19.937ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
13652025-04-30T06:26:19.937ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13662025-04-30T06:26:19.956ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13672025-04-30T06:26:19.956ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13682025-04-30T06:26:19.957ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13692025-04-30T06:26:19.960ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13702025-04-30T06:26:19.966ZDEBGtest_disk_expunge (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
13712025-04-30T06:26:19.966ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13722025-04-30T06:26:19.967ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
13732025-04-30T06:26:19.968ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13742025-04-30T06:26:19.973ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13752025-04-30T06:26:19.974ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13762025-04-30T06:26:19.974ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13772025-04-30T06:26:19.982ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13782025-04-30T06:26:19.983ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13792025-04-30T06:26:19.983ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13802025-04-30T06:26:19.984ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 984f2256-0303-4e62-b688-fe995471ffd2, lookup_type: ByName("oxide-service-pool") }
13812025-04-30T06:26:19.984ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13822025-04-30T06:26:19.984ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13832025-04-30T06:26:19.984ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
13842025-04-30T06:26:19.985ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13852025-04-30T06:26:19.990ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
13862025-04-30T06:26:19.990ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13872025-04-30T06:26:19.991ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13882025-04-30T06:26:19.991ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13892025-04-30T06:26:19.991ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13902025-04-30T06:26:19.993ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f64848bf-a723-4e00-ac35-b52602506493", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
13912025-04-30T06:26:19.993ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
13922025-04-30T06:26:19.993ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13932025-04-30T06:26:20.001ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 984f2256-0303-4e62-b688-fe995471ffd2, lookup_type: ByName("oxide-service-pool") } result = Ok(())
13942025-04-30T06:26:20.001ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13952025-04-30T06:26:20.001ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
13962025-04-30T06:26:20.001ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13972025-04-30T06:26:20.002ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13982025-04-30T06:26:20.002ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13992025-04-30T06:26:20.002ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14002025-04-30T06:26:20.003ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14012025-04-30T06:26:20.007ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14022025-04-30T06:26:20.007ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14032025-04-30T06:26:20.007ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14042025-04-30T06:26:20.007ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14052025-04-30T06:26:20.008ZTRACtest_disk_expunge (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 984f2256-0303-4e62-b688-fe995471ffd2, lookup_type: ByName("oxide-service-pool") }
14062025-04-30T06:26:20.008ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14072025-04-30T06:26:20.008ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14082025-04-30T06:26:20.008ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
14092025-04-30T06:26:20.009ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14102025-04-30T06:26:20.011ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14112025-04-30T06:26:20.018ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14122025-04-30T06:26:20.019ZTRACtest_disk_expunge (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
14132025-04-30T06:26:20.019ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
14142025-04-30T06:26:20.019ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
14152025-04-30T06:26:20.019ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14162025-04-30T06:26:20.019ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14172025-04-30T06:26:20.020ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14182025-04-30T06:26:20.022ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14192025-04-30T06:26:20.022ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14202025-04-30T06:26:20.022ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14212025-04-30T06:26:20.023ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14222025-04-30T06:26:20.027ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14232025-04-30T06:26:20.035ZDEBGtest_disk_expunge (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 984f2256-0303-4e62-b688-fe995471ffd2, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14242025-04-30T06:26:20.036ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc6e81a1-233f-4c09-8999-38cca0144f22", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
14252025-04-30T06:26:20.036ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
14262025-04-30T06:26:20.037ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14272025-04-30T06:26:20.037ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14282025-04-30T06:26:20.037ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14292025-04-30T06:26:20.037ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14302025-04-30T06:26:20.043ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14312025-04-30T06:26:20.050ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14322025-04-30T06:26:20.050ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14332025-04-30T06:26:20.050ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14342025-04-30T06:26:20.051ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14352025-04-30T06:26:20.051ZTRACtest_disk_expunge (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 984f2256-0303-4e62-b688-fe995471ffd2, lookup_type: ByName("oxide-service-pool") }
14362025-04-30T06:26:20.051ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14372025-04-30T06:26:20.051ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14382025-04-30T06:26:20.051ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
14392025-04-30T06:26:20.052ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14402025-04-30T06:26:20.059ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14412025-04-30T06:26:20.063ZDEBGtest_disk_expunge (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 984f2256-0303-4e62-b688-fe995471ffd2, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14422025-04-30T06:26:20.073ZTRACtest_disk_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
14432025-04-30T06:26:20.073ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14442025-04-30T06:26:20.073ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14452025-04-30T06:26:20.073ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
14462025-04-30T06:26:20.074ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14472025-04-30T06:26:20.075ZDEBGtest_disk_expunge (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
14482025-04-30T06:26:20.075ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet
14492025-04-30T06:26:20.075ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14502025-04-30T06:26:20.075ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14512025-04-30T06:26:20.075ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14522025-04-30T06:26:20.076ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14532025-04-30T06:26:20.078ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14542025-04-30T06:26:20.085ZDEBGtest_disk_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
14552025-04-30T06:26:20.087ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14562025-04-30T06:26:20.092ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
14572025-04-30T06:26:20.092ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14582025-04-30T06:26:20.092ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14592025-04-30T06:26:20.092ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14602025-04-30T06:26:20.139ZTRACtest_disk_expunge (service-firewall-rule-progator): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:06:08.248500Z, allowed_ips: None } background_task = service_firewall_rule_propagation
14612025-04-30T06:26:20.139ZDEBGtest_disk_expunge (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
14622025-04-30T06:26:20.139ZDEBGtest_disk_expunge (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
14632025-04-30T06:26:20.139ZDEBGtest_disk_expunge (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
14642025-04-30T06:26:20.165ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76da628c-aac6-403f-93fc-298aa752e1a7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
14652025-04-30T06:26:20.165ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
14662025-04-30T06:26:20.171ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2c27656-02eb-4e0d-91d7-88dfbb1201c2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
14672025-04-30T06:26:20.171ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
14682025-04-30T06:26:20.174ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17ddac79-a644-4b20-bdf8-4fb3477fc963", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:19 GMT"} })
14692025-04-30T06:26:20.175ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
14702025-04-30T06:26:20.182ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13fab568-752a-413b-a5cc-335fb14fb8c7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
14712025-04-30T06:26:20.182ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
14722025-04-30T06:26:20.185ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad0a7d9f-2a2a-44e4-b4bc-7f841a3757c5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
14732025-04-30T06:26:20.185ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
14742025-04-30T06:26:20.187ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfce670e-fc12-49eb-ad28-cedc77d81dcb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
14752025-04-30T06:26:20.187ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
14762025-04-30T06:26:20.189ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa949f19-db40-42d9-94dd-71fde9ca6e33", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
14772025-04-30T06:26:20.190ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
14782025-04-30T06:26:20.191ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6ba0f3e-31ed-4fd9-9d17-1aeacbe3792e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
14792025-04-30T06:26:20.191ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
14802025-04-30T06:26:20.253ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "15bb9243-5cef-4d2a-851d-b14ae6976754", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
14812025-04-30T06:26:20.253ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
14822025-04-30T06:26:20.285ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65ff5b00-e44a-4e53-9696-5604be133caa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
14832025-04-30T06:26:20.285ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
14842025-04-30T06:26:20.287ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9eda4201-cce9-4fd6-8918-9eafce30f31f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
14852025-04-30T06:26:20.287ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
14862025-04-30T06:26:20.288ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cbf3107-5527-44a3-85f6-72dee1d61803", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
14872025-04-30T06:26:20.288ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
14882025-04-30T06:26:20.305ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "136b5b08-637e-453d-8195-92edb6d6d85e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
14892025-04-30T06:26:20.305ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
14902025-04-30T06:26:20.306ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dde2fcd2-ec16-4118-9315-03034d97d5d2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
14912025-04-30T06:26:20.306ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
14922025-04-30T06:26:20.316ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a2f1f6b-4fc5-4a17-a6e0-3c3fba73e90a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
14932025-04-30T06:26:20.319ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
14942025-04-30T06:26:20.335ZDEBGtest_disk_expunge (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
14952025-04-30T06:26:20.346ZDEBGtest_disk_expunge (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
14962025-04-30T06:26:20.346ZINFOtest_disk_expunge (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation
14972025-04-30T06:26:20.346ZDEBGtest_disk_expunge (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 5.602476047s
14982025-04-30T06:26:20.346ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 5.602581867s iteration = 1 status = {}
14992025-04-30T06:26:20.349ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e189d509-32e3-4b71-b920-868e842325f0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15002025-04-30T06:26:20.349ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
15012025-04-30T06:26:20.350ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cc9583a-c882-4dad-9967-d44f6b0a0e67", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15022025-04-30T06:26:20.350ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
15032025-04-30T06:26:20.353ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dbdb7ff-c9b3-46e4-a0e0-ebaf4bdf4d4a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15042025-04-30T06:26:20.353ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
15052025-04-30T06:26:20.356ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "981d7d8c-7782-43dc-a43e-e14e834da744", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15062025-04-30T06:26:20.356ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
15072025-04-30T06:26:20.360ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b78dce3e-b426-4c47-917d-8e143f3640c3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15082025-04-30T06:26:20.360ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
15092025-04-30T06:26:20.364ZINFOtest_disk_expunge (InternalApi): inserted blueprint blueprint_id = ecb5bbc1-0292-4ab5-8dfc-26579f1d65a3
15102025-04-30T06:26:20.364ZTRACtest_disk_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
15112025-04-30T06:26:20.364ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15122025-04-30T06:26:20.364ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15132025-04-30T06:26:20.364ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
15142025-04-30T06:26:20.365ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15152025-04-30T06:26:20.365ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6605330e-a760-448a-91f7-cea9912db7e6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15162025-04-30T06:26:20.366ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
15172025-04-30T06:26:20.370ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15182025-04-30T06:26:20.374ZDEBGtest_disk_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
15192025-04-30T06:26:20.374ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0bc3652-88bf-4f75-929f-8b05b55fb949", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15202025-04-30T06:26:20.374ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
15212025-04-30T06:26:20.378ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "141d5f2a-54d9-4e4e-b405-35b5cce0a4ed", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15222025-04-30T06:26:20.378ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
15232025-04-30T06:26:20.379ZINFOtest_disk_expunge (InternalApi): No networking records needed for crucible_pantry service
15242025-04-30T06:26:20.380ZINFOtest_disk_expunge (InternalApi): No networking records needed for internal_dns service
15252025-04-30T06:26:20.380ZINFOtest_disk_expunge (InternalApi): No networking records needed for cockroach_db service
15262025-04-30T06:26:20.380ZINFOtest_disk_expunge (InternalApi): No networking records needed for clickhouse service
15272025-04-30T06:26:20.387ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9df4b452-9fac-468f-b968-9d3fa6978f17", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15282025-04-30T06:26:20.387ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
15292025-04-30T06:26:20.388ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d44a8cd-0fcb-4416-b372-cc604c5285f6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15302025-04-30T06:26:20.388ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
15312025-04-30T06:26:20.389ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e7ee63b-a85c-4297-bd31-8cab97406141", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15322025-04-30T06:26:20.389ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
15332025-04-30T06:26:20.390ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e3acf88-9f9c-4ddf-9350-3cc3a9c0b82c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15342025-04-30T06:26:20.390ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
15352025-04-30T06:26:20.392ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbd43a8d-3289-45e6-aac4-55fb75b3459b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15362025-04-30T06:26:20.393ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
15372025-04-30T06:26:20.402ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1a4adf5-69a7-45aa-9025-bdf4d3379353", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15382025-04-30T06:26:20.402ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
15392025-04-30T06:26:20.428ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16dc8ec1-40aa-4c11-9e37-3ea9670e1aa7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15402025-04-30T06:26:20.428ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
15412025-04-30T06:26:20.431ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45b2ab62-d15a-4e91-b420-65066026f2c4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15422025-04-30T06:26:20.431ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
15432025-04-30T06:26:20.435ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67aad5d3-32cb-47cc-b326-32e1561bdd34", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15442025-04-30T06:26:20.435ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
15452025-04-30T06:26:20.437ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "245c272e-3dd7-44b6-8a3b-a22e7cbfaf33", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15462025-04-30T06:26:20.437ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
15472025-04-30T06:26:20.442ZINFOtest_disk_expunge (InternalApi): Inserted networking records for external_dns service
15482025-04-30T06:26:20.442ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a02f58d6-f26e-4766-a4c0-51fb8aa53de7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15492025-04-30T06:26:20.443ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
15502025-04-30T06:26:20.445ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f117dacc-4961-4a2a-b099-18da48396ed2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15512025-04-30T06:26:20.445ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
15522025-04-30T06:26:20.446ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58aff4ea-76ad-458f-bdcd-6833fdc46d3b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15532025-04-30T06:26:20.446ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
15542025-04-30T06:26:20.447ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6ca8977-a33e-4683-b31d-220b07216892", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15552025-04-30T06:26:20.447ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
15562025-04-30T06:26:20.448ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eeed06af-cf6b-4b01-906c-8b67d59a1548", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15572025-04-30T06:26:20.449ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
15582025-04-30T06:26:20.449ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5aae62c7-12d5-4d79-b0ad-1d6c229aa8de", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15592025-04-30T06:26:20.450ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15602025-04-30T06:26:20.450ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15612025-04-30T06:26:20.450ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15622025-04-30T06:26:20.450ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15632025-04-30T06:26:20.451ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15642025-04-30T06:26:20.453ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15652025-04-30T06:26:20.458ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15662025-04-30T06:26:20.459ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15672025-04-30T06:26:20.459ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15682025-04-30T06:26:20.459ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15692025-04-30T06:26:20.459ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15702025-04-30T06:26:20.459ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15712025-04-30T06:26:20.480ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15722025-04-30T06:26:20.485ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15732025-04-30T06:26:20.485ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15742025-04-30T06:26:20.486ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15752025-04-30T06:26:20.486ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15762025-04-30T06:26:20.488ZINFOtest_disk_expunge (InternalApi): Inserted networking records for nexus service
15772025-04-30T06:26:20.488ZINFOtest_disk_expunge (InternalApi): Inserted service networking records
15782025-04-30T06:26:20.488ZINFOtest_disk_expunge (InternalApi): Inserted physical disks
15792025-04-30T06:26:20.488ZINFOtest_disk_expunge (InternalApi): Inserted zpools
15802025-04-30T06:26:20.488ZINFOtest_disk_expunge (InternalApi): Inserted datasets
15812025-04-30T06:26:20.558ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
15822025-04-30T06:26:20.558ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
15832025-04-30T06:26:20.560ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:53079 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
15842025-04-30T06:26:20.561ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:53079 req_id = 7e713bc2-ec25-449a-ad36-786a17949e02 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15852025-04-30T06:26:20.564ZINFOtest_disk_expunge (dropshot): request completed latency_us = 3228 local_addr = [::1]:44559 method = GET remote_addr = [::1]:53079 req_id = 7e713bc2-ec25-449a-ad36-786a17949e02 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15862025-04-30T06:26:20.564ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e713bc2-ec25-449a-ad36-786a17949e02", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:26:20 GMT"} })
15872025-04-30T06:26:20.567ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15882025-04-30T06:26:20.567ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15892025-04-30T06:26:20.567ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
15902025-04-30T06:26:20.568ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
15912025-04-30T06:26:20.568ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15922025-04-30T06:26:20.568ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15932025-04-30T06:26:20.568ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15942025-04-30T06:26:20.569ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15952025-04-30T06:26:20.573ZINFOtest_disk_expunge (InternalApi): Populated DNS tables for internal DNS
15962025-04-30T06:26:20.576ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15972025-04-30T06:26:20.582ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
15982025-04-30T06:26:20.582ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15992025-04-30T06:26:20.582ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
16002025-04-30T06:26:20.586ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
16012025-04-30T06:26:20.588ZINFOtest_disk_expunge (InternalApi): Populated DNS tables for external DNS
16022025-04-30T06:26:20.588ZTRACtest_disk_expunge (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16032025-04-30T06:26:20.588ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16042025-04-30T06:26:20.588ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16052025-04-30T06:26:20.588ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16062025-04-30T06:26:20.589ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16072025-04-30T06:26:20.592ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16082025-04-30T06:26:20.602ZDEBGtest_disk_expunge (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16092025-04-30T06:26:20.602ZTRACtest_disk_expunge (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16102025-04-30T06:26:20.602ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16112025-04-30T06:26:20.602ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16122025-04-30T06:26:20.602ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16132025-04-30T06:26:20.602ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16142025-04-30T06:26:20.611ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16152025-04-30T06:26:20.615ZDEBGtest_disk_expunge (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16162025-04-30T06:26:20.654ZTRACtest_disk_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16172025-04-30T06:26:20.654ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16182025-04-30T06:26:20.654ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16192025-04-30T06:26:20.654ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16202025-04-30T06:26:20.655ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16212025-04-30T06:26:20.667ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16222025-04-30T06:26:20.676ZDEBGtest_disk_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16232025-04-30T06:26:20.676ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16242025-04-30T06:26:20.676ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16252025-04-30T06:26:20.676ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16262025-04-30T06:26:20.676ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16272025-04-30T06:26:20.676ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16282025-04-30T06:26:20.686ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16292025-04-30T06:26:20.691ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16302025-04-30T06:26:20.698ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16312025-04-30T06:26:20.698ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16322025-04-30T06:26:20.698ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16332025-04-30T06:26:20.698ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16342025-04-30T06:26:20.699ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16352025-04-30T06:26:20.711ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16362025-04-30T06:26:20.718ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16372025-04-30T06:26:20.736ZINFOtest_disk_expunge (InternalApi): Created recovery silo
16382025-04-30T06:26:20.738ZINFOtest_disk_expunge (InternalApi): Created recovery user
16392025-04-30T06:26:20.748ZINFOtest_disk_expunge (InternalApi): Created recovery user's password
16402025-04-30T06:26:20.748ZTRACtest_disk_expunge (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 5c399561-4757-4f96-ac4f-8e23573dcd26, lookup_type: ById(5c399561-4757-4f96-ac4f-8e23573dcd26) }
16412025-04-30T06:26:20.748ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 5c399561-4757-4f96-ac4f-8e23573dcd26 resource_type = Silo
16422025-04-30T06:26:20.748ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16432025-04-30T06:26:20.748ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16442025-04-30T06:26:20.749ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16452025-04-30T06:26:20.755ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16462025-04-30T06:26:20.755ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16472025-04-30T06:26:20.755ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16482025-04-30T06:26:20.756ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16492025-04-30T06:26:20.772ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16502025-04-30T06:26:20.776ZDEBGtest_disk_expunge (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 5c399561-4757-4f96-ac4f-8e23573dcd26, lookup_type: ById(5c399561-4757-4f96-ac4f-8e23573dcd26) } result = Ok(())
16512025-04-30T06:26:20.776ZDEBGtest_disk_expunge (InternalApi): Generated role assignment queries
16522025-04-30T06:26:20.781ZINFOtest_disk_expunge (InternalApi): Granted Silo privileges
16532025-04-30T06:26:20.781ZTRACtest_disk_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16542025-04-30T06:26:20.781ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16552025-04-30T06:26:20.781ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16562025-04-30T06:26:20.782ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16572025-04-30T06:26:20.783ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16582025-04-30T06:26:20.787ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16592025-04-30T06:26:20.791ZDEBGtest_disk_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16602025-04-30T06:26:20.807ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16612025-04-30T06:26:20.807ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16622025-04-30T06:26:20.809ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16632025-04-30T06:26:20.814ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16642025-04-30T06:26:20.814ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16652025-04-30T06:26:20.815ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16662025-04-30T06:26:20.819ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16672025-04-30T06:26:20.819ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16682025-04-30T06:26:20.819ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16692025-04-30T06:26:20.824ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16702025-04-30T06:26:20.825ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16712025-04-30T06:26:20.825ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16722025-04-30T06:26:20.825ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16732025-04-30T06:26:20.826ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16742025-04-30T06:26:20.827ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16752025-04-30T06:26:20.828ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16762025-04-30T06:26:20.828ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16772025-04-30T06:26:20.837ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16782025-04-30T06:26:20.877ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16792025-04-30T06:26:20.878ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16802025-04-30T06:26:20.878ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16812025-04-30T06:26:20.882ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16822025-04-30T06:26:20.885ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16832025-04-30T06:26:20.894ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
16842025-04-30T06:26:20.894ZTRACtest_disk_expunge (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
16852025-04-30T06:26:20.894ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16862025-04-30T06:26:20.894ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16872025-04-30T06:26:20.895ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16882025-04-30T06:26:20.895ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16892025-04-30T06:26:20.896ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16902025-04-30T06:26:20.901ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16912025-04-30T06:26:20.902ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16922025-04-30T06:26:20.902ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16932025-04-30T06:26:20.902ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16942025-04-30T06:26:20.908ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16952025-04-30T06:26:20.908ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16962025-04-30T06:26:20.908ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
16972025-04-30T06:26:20.909ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16982025-04-30T06:26:20.912ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16992025-04-30T06:26:20.922ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17002025-04-30T06:26:20.922ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17012025-04-30T06:26:20.922ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17022025-04-30T06:26:20.924ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17032025-04-30T06:26:20.928ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17042025-04-30T06:26:20.928ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17052025-04-30T06:26:20.929ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17062025-04-30T06:26:20.935ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17072025-04-30T06:26:20.935ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17082025-04-30T06:26:20.936ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17092025-04-30T06:26:20.939ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17102025-04-30T06:26:20.939ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17112025-04-30T06:26:20.940ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17122025-04-30T06:26:20.943ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
17132025-04-30T06:26:20.943ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17142025-04-30T06:26:20.943ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17152025-04-30T06:26:20.944ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17162025-04-30T06:26:20.944ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17172025-04-30T06:26:20.948ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17182025-04-30T06:26:20.948ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17192025-04-30T06:26:20.948ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17202025-04-30T06:26:20.949ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17212025-04-30T06:26:20.956ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17222025-04-30T06:26:20.956ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17232025-04-30T06:26:20.956ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17242025-04-30T06:26:20.957ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17252025-04-30T06:26:20.960ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17262025-04-30T06:26:20.967ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17272025-04-30T06:26:20.967ZTRACtest_disk_expunge (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
17282025-04-30T06:26:20.967ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17292025-04-30T06:26:20.967ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17302025-04-30T06:26:20.968ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17312025-04-30T06:26:20.971ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17322025-04-30T06:26:20.971ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17332025-04-30T06:26:20.972ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17342025-04-30T06:26:20.975ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17352025-04-30T06:26:20.975ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17362025-04-30T06:26:20.976ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17372025-04-30T06:26:20.979ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17382025-04-30T06:26:20.979ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17392025-04-30T06:26:20.980ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17402025-04-30T06:26:20.988ZTRACtest_disk_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17412025-04-30T06:26:20.988ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17422025-04-30T06:26:20.988ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17432025-04-30T06:26:20.988ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17442025-04-30T06:26:20.989ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17452025-04-30T06:26:20.993ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17462025-04-30T06:26:20.994ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17472025-04-30T06:26:20.994ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17482025-04-30T06:26:20.995ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17492025-04-30T06:26:21.001ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17502025-04-30T06:26:21.002ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17512025-04-30T06:26:21.008ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17522025-04-30T06:26:21.012ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17532025-04-30T06:26:21.016ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17542025-04-30T06:26:21.028ZDEBGtest_disk_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17552025-04-30T06:26:21.028ZTRACtest_disk_expunge (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
17562025-04-30T06:26:21.028ZTRACtest_disk_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17572025-04-30T06:26:21.028ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17582025-04-30T06:26:21.028ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17592025-04-30T06:26:21.028ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17602025-04-30T06:26:21.028ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17612025-04-30T06:26:21.032ZINFOtest_disk_expunge (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 5af26b40-5399-4991-864a-dbacb877f9ff
17622025-04-30T06:26:21.032ZDEBGtest_disk_expunge (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 5af26b40-5399-4991-864a-dbacb877f9ff time_started = 2025-04-30 06:26:19.371617 UTC
17632025-04-30T06:26:21.032ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 5.138847478s iteration = 2 status = {"collection_id":"5af26b40-5399-4991-864a-dbacb877f9ff","time_done":"2025-04-30 06:26:20.567220 UTC","time_started":"2025-04-30 06:26:19.371617 UTC"}
17642025-04-30T06:26:21.032ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
17652025-04-30T06:26:21.032ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 45.851\u{b5}s iteration = 2 status = {"error":"task disabled"}
17662025-04-30T06:26:21.032ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
17672025-04-30T06:26:21.032ZWARNtest_disk_expunge (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
17682025-04-30T06:26:21.032ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 15.972\u{b5}s iteration = 2 status = {"error":"no blueprint"}
17692025-04-30T06:26:21.033ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17702025-04-30T06:26:21.033ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17712025-04-30T06:26:21.033ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17722025-04-30T06:26:21.033ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17732025-04-30T06:26:21.039ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17742025-04-30T06:26:21.039ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17752025-04-30T06:26:21.040ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17762025-04-30T06:26:21.040ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17772025-04-30T06:26:21.049ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17782025-04-30T06:26:21.059ZDEBGtest_disk_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17792025-04-30T06:26:21.059ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17802025-04-30T06:26:21.059ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17812025-04-30T06:26:21.059ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17822025-04-30T06:26:21.073ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17832025-04-30T06:26:21.074ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17842025-04-30T06:26:21.074ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17852025-04-30T06:26:21.086ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17862025-04-30T06:26:21.089ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17872025-04-30T06:26:21.090ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17882025-04-30T06:26:21.102ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17892025-04-30T06:26:21.102ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17902025-04-30T06:26:21.103ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17912025-04-30T06:26:21.106ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17922025-04-30T06:26:21.107ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17932025-04-30T06:26:21.107ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17942025-04-30T06:26:21.110ZTRACtest_disk_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
17952025-04-30T06:26:21.110ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17962025-04-30T06:26:21.110ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17972025-04-30T06:26:21.110ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
17982025-04-30T06:26:21.110ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17992025-04-30T06:26:21.114ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18002025-04-30T06:26:21.114ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18012025-04-30T06:26:21.114ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18022025-04-30T06:26:21.115ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18032025-04-30T06:26:21.119ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18042025-04-30T06:26:21.119ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18052025-04-30T06:26:21.119ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18062025-04-30T06:26:21.120ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18072025-04-30T06:26:21.125ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18082025-04-30T06:26:21.135ZDEBGtest_disk_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18092025-04-30T06:26:21.136ZTRACtest_disk_expunge (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
18102025-04-30T06:26:21.136ZTRACtest_disk_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
18112025-04-30T06:26:21.136ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
18122025-04-30T06:26:21.136ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18132025-04-30T06:26:21.136ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18142025-04-30T06:26:21.136ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18152025-04-30T06:26:21.139ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18162025-04-30T06:26:21.139ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18172025-04-30T06:26:21.139ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18182025-04-30T06:26:21.139ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18192025-04-30T06:26:21.141ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18202025-04-30T06:26:21.141ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18212025-04-30T06:26:21.141ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18222025-04-30T06:26:21.142ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18232025-04-30T06:26:21.144ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18242025-04-30T06:26:21.151ZDEBGtest_disk_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18252025-04-30T06:26:21.152ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18262025-04-30T06:26:21.152ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18272025-04-30T06:26:21.152ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18282025-04-30T06:26:21.159ZDEBGtest_disk_expunge (nexus): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: dbdfd5b5-8b4b-474d-98e3-880e650dc7ed, kind: Service { id: 68c89553-4ad9-4464-83e9-6636df41c29b }, name: Name("external-dns-68c89553-4ad9-4464-83e9-6636df41c29b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: 76b6444e-557a-4565-9b1f-2d99179e79ac, kind: Service { id: 78121327-166d-4e33-935c-82ab8635bf33 }, name: Name("nexus-78121327-166d-4e33-935c-82ab8635bf33"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
18292025-04-30T06:26:21.159ZTRACtest_disk_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
18302025-04-30T06:26:21.159ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18312025-04-30T06:26:21.159ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18322025-04-30T06:26:21.159ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18332025-04-30T06:26:21.160ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18342025-04-30T06:26:21.162ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18352025-04-30T06:26:21.168ZDEBGtest_disk_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
18362025-04-30T06:26:21.168ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18372025-04-30T06:26:21.168ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
18382025-04-30T06:26:21.168ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18392025-04-30T06:26:21.176ZTRACtest_disk_expunge (nexus): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:26:20.791586Z, allowed_ips: None }
18402025-04-30T06:26:21.176ZDEBGtest_disk_expunge (nexus): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
18412025-04-30T06:26:21.177ZDEBGtest_disk_expunge (nexus): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: dbdfd5b5-8b4b-474d-98e3-880e650dc7ed, kind: Service { id: 68c89553-4ad9-4464-83e9-6636df41c29b }, name: Name("external-dns-68c89553-4ad9-4464-83e9-6636df41c29b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 76b6444e-557a-4565-9b1f-2d99179e79ac, kind: Service { id: 78121327-166d-4e33-935c-82ab8635bf33 }, name: Name("nexus-78121327-166d-4e33-935c-82ab8635bf33"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
18422025-04-30T06:26:21.177ZDEBGtest_disk_expunge (nexus): resolved 2 rules for sleds
18432025-04-30T06:26:21.339ZDEBGtest_disk_expunge (nexus): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:26:14.842443Z, time_modified: 2025-04-30T06:26:14.842443Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(44559), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(47123) }]
18442025-04-30T06:26:21.339ZDEBGtest_disk_expunge (nexus): sending firewall rules to sled agents
18452025-04-30T06:26:21.339ZTRACtest_disk_expunge (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18462025-04-30T06:26:21.339ZDEBGtest_disk_expunge (InstanceAllocator): roles roles = RoleSet { roles: {} }
18472025-04-30T06:26:21.339ZDEBGtest_disk_expunge (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18482025-04-30T06:26:21.343ZTRACtest_disk_expunge (InstanceAllocator): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
18492025-04-30T06:26:21.343ZTRACtest_disk_expunge (InstanceAllocator): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18502025-04-30T06:26:21.343ZTRACtest_disk_expunge (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18512025-04-30T06:26:21.343ZDEBGtest_disk_expunge (InstanceAllocator): roles roles = RoleSet { roles: {} }
18522025-04-30T06:26:21.344ZDEBGtest_disk_expunge (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18532025-04-30T06:26:21.346ZDEBGtest_disk_expunge (InstanceAllocator): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "viewer")} }
18542025-04-30T06:26:21.350ZDEBGtest_disk_expunge (InstanceAllocator): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
18552025-04-30T06:26:21.394ZDEBGtest_disk_expunge (nexus): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:44559/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18562025-04-30T06:26:21.395ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:62459 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
18572025-04-30T06:26:21.395ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = PUT remote_addr = [::1]:62459 req_id = ba1e450a-bad5-4300-b598-80a453821ad7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18582025-04-30T06:26:21.397ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1395 local_addr = [::1]:44559 method = PUT remote_addr = [::1]:62459 req_id = ba1e450a-bad5-4300-b598-80a453821ad7 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18592025-04-30T06:26:21.397ZDEBGtest_disk_expunge (nexus): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:44559/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "ba1e450a-bad5-4300-b598-80a453821ad7", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
18602025-04-30T06:26:21.398ZINFOtest_disk_expunge (nexus): updated firewall rules on 1 sleds
18612025-04-30T06:26:21.398ZTRACtest_disk_expunge (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
18622025-04-30T06:26:21.398ZDEBGtest_disk_expunge (ServiceBalancer): roles roles = RoleSet { roles: {} }
18632025-04-30T06:26:21.399ZDEBGtest_disk_expunge (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
18642025-04-30T06:26:21.400ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Signaled
18652025-04-30T06:26:21.400ZDEBGtest_disk_expunge (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18662025-04-30T06:26:21.400ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
18672025-04-30T06:26:21.400ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18682025-04-30T06:26:21.400ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18692025-04-30T06:26:21.400ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Signaled
18702025-04-30T06:26:21.400ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
18712025-04-30T06:26:21.401ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Signaled
18722025-04-30T06:26:21.401ZDEBGtest_disk_expunge (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18732025-04-30T06:26:21.401ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
18742025-04-30T06:26:21.401ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18752025-04-30T06:26:21.401ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18762025-04-30T06:26:21.401ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Signaled
18772025-04-30T06:26:21.401ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
18782025-04-30T06:26:21.401ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Signaled
18792025-04-30T06:26:21.401ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
18802025-04-30T06:26:21.401ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18812025-04-30T06:26:21.401ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
18822025-04-30T06:26:21.401ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18832025-04-30T06:26:21.402ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18842025-04-30T06:26:21.402ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 3 reason = Signaled
18852025-04-30T06:26:21.402ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
18862025-04-30T06:26:21.402ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18872025-04-30T06:26:21.402ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
18882025-04-30T06:26:21.402ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
18892025-04-30T06:26:21.403ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
18902025-04-30T06:26:21.404ZTRACtest_disk_expunge (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 34 3b 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 4;..........._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:58295 req_id = ead9d97d-a657-4110-bca8-089a70a9241f
18912025-04-30T06:26:21.404ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13371,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58295 req_id = ead9d97d-a657-4110-bca8-089a70a9241f
18922025-04-30T06:26:21.404ZDEBGtest_disk_expunge (store): query key key = _nameservice._tcp
18932025-04-30T06:26:21.404ZDEBGtest_disk_expunge (store): query key key = 2784c366-c956-4ffa-afcb-17669db9c967.host
18942025-04-30T06:26:21.404ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("2784c366-c956-4ffa-afcb-17669db9c967.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58295 query = LowerQuery { name: LowerName(Name("_nameservice._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36493, target: Name("2784c366-c956-4ffa-afcb-17669db9c967.host.control-plane.oxide.internal") })) }] req_id = ead9d97d-a657-4110-bca8-089a70a9241f
18952025-04-30T06:26:21.404ZTRACtest_disk_expunge (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: e3 c4 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59828 req_id = b4a53e9b-f5f9-4e35-81ae-74e934f8a004
18962025-04-30T06:26:21.404ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58308,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59828 req_id = b4a53e9b-f5f9-4e35-81ae-74e934f8a004
18972025-04-30T06:26:21.405ZDEBGtest_disk_expunge (store): query key key = _external-dns._tcp
18982025-04-30T06:26:21.405ZDEBGtest_disk_expunge (store): query key key = 68c89553-4ad9-4464-83e9-6636df41c29b.host
18992025-04-30T06:26:21.405ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("68c89553-4ad9-4464-83e9-6636df41c29b.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59828 query = LowerQuery { name: LowerName(Name("_external-dns._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 52024, target: Name("68c89553-4ad9-4464-83e9-6636df41c29b.host.control-plane.oxide.internal") })) }] req_id = b4a53e9b-f5f9-4e35-81ae-74e934f8a004
19002025-04-30T06:26:21.406ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36493, target: Name("2784c366-c956-4ffa-afcb-17669db9c967.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("2784c366-c956-4ffa-afcb-17669db9c967.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3246, tv_nsec: 205203231 } })
19012025-04-30T06:26:21.406ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
19022025-04-30T06:26:21.406ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 52024, target: Name("68c89553-4ad9-4464-83e9-6636df41c29b.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("68c89553-4ad9-4464-83e9-6636df41c29b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3246, tv_nsec: 205527397 } })
19032025-04-30T06:26:21.406ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
19042025-04-30T06:26:21.407ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19052025-04-30T06:26:21.407ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19062025-04-30T06:26:21.407ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19072025-04-30T06:26:21.407ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19082025-04-30T06:26:21.408ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19092025-04-30T06:26:21.408ZTRACtest_disk_expunge (ServiceBalancer): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
19102025-04-30T06:26:21.408ZTRACtest_disk_expunge (ServiceBalancer): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19112025-04-30T06:26:21.408ZTRACtest_disk_expunge (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
19122025-04-30T06:26:21.408ZDEBGtest_disk_expunge (ServiceBalancer): roles roles = RoleSet { roles: {} }
19132025-04-30T06:26:21.409ZDEBGtest_disk_expunge (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
19142025-04-30T06:26:21.409ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19152025-04-30T06:26:21.410ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19162025-04-30T06:26:21.410ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19172025-04-30T06:26:21.410ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19182025-04-30T06:26:21.411ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19192025-04-30T06:26:21.411ZTRACtest_disk_expunge (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 1d 8c 01 00 00 01 00 00 00 00 00 01 24 32 37 38 ............$278\n0010: 34 63 33 36 36 2d 63 39 35 36 2d 34 66 66 61 2d 4c366-c956-4ffa-\n0020: 61 66 63 62 2d 31 37 36 36 39 64 62 39 63 39 36 afcb-17669db9c96\n0030: 37 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 7.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:49569 req_id = cb07a595-a9f0-4ac8-92e4-6576f01bd12e
19202025-04-30T06:26:21.411ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7564,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("2784c366-c956-4ffa-afcb-17669db9c967.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("2784c366-c956-4ffa-afcb-17669db9c967.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 50,\n 55,\n 56,\n 52,\n 99,\n 51,\n 54,\n 54,\n 45,\n 99,\n 57,\n 53,\n 54,\n 45,\n 52,\n 102,\n 102,\n 97,\n 45,\n 97,\n 102,\n 99,\n 98,\n 45,\n 49,\n 55,\n 54,\n 54,\n 57,\n 100,\n 98,\n 57,\n 99,\n 57,\n 54,\n 55,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49569 req_id = cb07a595-a9f0-4ac8-92e4-6576f01bd12e
19212025-04-30T06:26:21.411ZDEBGtest_disk_expunge (store): query key key = 2784c366-c956-4ffa-afcb-17669db9c967.host
19222025-04-30T06:26:21.411ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49569 query = LowerQuery { name: LowerName(Name("2784c366-c956-4ffa-afcb-17669db9c967.host.control-plane.oxide.internal.")), original: Query { name: Name("2784c366-c956-4ffa-afcb-17669db9c967.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("2784c366-c956-4ffa-afcb-17669db9c967.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cb07a595-a9f0-4ac8-92e4-6576f01bd12e
19232025-04-30T06:26:21.412ZTRACtest_disk_expunge (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 52 07 01 00 00 01 00 00 00 00 00 01 24 36 38 63 R...........$68c\n0010: 38 39 35 35 33 2d 34 61 64 39 2d 34 34 36 34 2d 89553-4ad9-4464-\n0020: 38 33 65 39 2d 36 36 33 36 64 66 34 31 63 32 39 83e9-6636df41c29\n0030: 62 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 b.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:62594 req_id = 43aed822-2e4a-4513-82e4-7f0b08bff3e0
19242025-04-30T06:26:21.412ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20999,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("68c89553-4ad9-4464-83e9-6636df41c29b.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("68c89553-4ad9-4464-83e9-6636df41c29b.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 54,\n 56,\n 99,\n 56,\n 57,\n 53,\n 53,\n 51,\n 45,\n 52,\n 97,\n 100,\n 57,\n 45,\n 52,\n 52,\n 54,\n 52,\n 45,\n 56,\n 51,\n 101,\n 57,\n 45,\n 54,\n 54,\n 51,\n 54,\n 100,\n 102,\n 52,\n 49,\n 99,\n 50,\n 57,\n 98,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62594 req_id = 43aed822-2e4a-4513-82e4-7f0b08bff3e0
19252025-04-30T06:26:21.412ZDEBGtest_disk_expunge (store): query key key = 68c89553-4ad9-4464-83e9-6636df41c29b.host
19262025-04-30T06:26:21.412ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62594 query = LowerQuery { name: LowerName(Name("68c89553-4ad9-4464-83e9-6636df41c29b.host.control-plane.oxide.internal.")), original: Query { name: Name("68c89553-4ad9-4464-83e9-6636df41c29b.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("68c89553-4ad9-4464-83e9-6636df41c29b.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 43aed822-2e4a-4513-82e4-7f0b08bff3e0
19272025-04-30T06:26:21.413ZINFOtest_disk_expunge (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:36493] } background_task = dns_servers_internal dns_group = internal
19282025-04-30T06:26:21.413ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 12.729292ms iteration = 2 status = {"addresses":["[::1]:36493"]}
19292025-04-30T06:26:21.413ZINFOtest_disk_expunge (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:52024] } background_task = dns_servers_external dns_group = external
19302025-04-30T06:26:21.413ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 12.178716ms iteration = 2 status = {"addresses":["[::1]:52024"]}
19312025-04-30T06:26:21.413ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
19322025-04-30T06:26:21.413ZWARNtest_disk_expunge (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config
19332025-04-30T06:26:21.413ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 19.77\u{b5}s iteration = 2 status = {"error":"no config"}
19342025-04-30T06:26:21.413ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
19352025-04-30T06:26:21.414ZWARNtest_disk_expunge (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config
19362025-04-30T06:26:21.414ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 10.561\u{b5}s iteration = 2 status = {"error":"no config"}
19372025-04-30T06:26:21.416ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19382025-04-30T06:26:21.431ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19392025-04-30T06:26:21.431ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19402025-04-30T06:26:21.431ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19412025-04-30T06:26:21.431ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19422025-04-30T06:26:21.432ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19432025-04-30T06:26:21.436ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
19442025-04-30T06:26:21.436ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19452025-04-30T06:26:21.436ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19462025-04-30T06:26:21.436ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19472025-04-30T06:26:21.436ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19482025-04-30T06:26:21.442ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19492025-04-30T06:26:21.443ZDEBGtest_disk_expunge (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19502025-04-30T06:26:21.443ZDEBGtest_disk_expunge (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19512025-04-30T06:26:21.443ZDEBGtest_disk_expunge (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
19522025-04-30T06:26:21.443ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19532025-04-30T06:26:21.443ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19542025-04-30T06:26:21.443ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19552025-04-30T06:26:21.443ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19562025-04-30T06:26:21.444ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19572025-04-30T06:26:21.444ZDEBGtest_disk_expunge (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19582025-04-30T06:26:21.450ZDEBGtest_disk_expunge (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
19592025-04-30T06:26:21.450ZINFOtest_disk_expunge (nexus): Rack initialized
19602025-04-30T06:26:21.450ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19612025-04-30T06:26:21.450ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
19622025-04-30T06:26:21.450ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19632025-04-30T06:26:21.450ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19642025-04-30T06:26:21.455ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19652025-04-30T06:26:21.456ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
19662025-04-30T06:26:21.456ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19672025-04-30T06:26:21.456ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19682025-04-30T06:26:21.456ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19692025-04-30T06:26:21.457ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19702025-04-30T06:26:21.457ZDEBGtest_disk_expunge (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19712025-04-30T06:26:21.457ZDEBGtest_disk_expunge (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19722025-04-30T06:26:21.457ZDEBGtest_disk_expunge (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19732025-04-30T06:26:21.457ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19742025-04-30T06:26:21.457ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19752025-04-30T06:26:21.457ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19762025-04-30T06:26:21.457ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19772025-04-30T06:26:21.458ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19782025-04-30T06:26:21.458ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19792025-04-30T06:26:21.485ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19802025-04-30T06:26:21.485ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19812025-04-30T06:26:21.485ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19822025-04-30T06:26:21.486ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19832025-04-30T06:26:21.486ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
19842025-04-30T06:26:21.486ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
19852025-04-30T06:26:21.487ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: bc 98 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59430 req_id = acc97a6d-2b68-43bb-aa1a-efbdeeaf66bf
19862025-04-30T06:26:21.487ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48280,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59430 req_id = acc97a6d-2b68-43bb-aa1a-efbdeeaf66bf
19872025-04-30T06:26:21.487ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
19882025-04-30T06:26:21.488ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
19892025-04-30T06:26:21.488ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59430 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = acc97a6d-2b68-43bb-aa1a-efbdeeaf66bf
19902025-04-30T06:26:21.489ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3246, tv_nsec: 287939518 } })
19912025-04-30T06:26:21.489ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
19922025-04-30T06:26:21.490ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19932025-04-30T06:26:21.497ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19942025-04-30T06:26:21.497ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19952025-04-30T06:26:21.497ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19962025-04-30T06:26:21.497ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19972025-04-30T06:26:21.498ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19982025-04-30T06:26:21.498ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
19992025-04-30T06:26:21.498ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20002025-04-30T06:26:21.499ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 83 cb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55830 req_id = 85e0c4e3-e809-4f8c-abbb-e668f5c31771
20012025-04-30T06:26:21.499ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33739,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55830 req_id = 85e0c4e3-e809-4f8c-abbb-e668f5c31771
20022025-04-30T06:26:21.499ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
20032025-04-30T06:26:21.499ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55830 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 85e0c4e3-e809-4f8c-abbb-e668f5c31771
20042025-04-30T06:26:21.597ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
20052025-04-30T06:26:21.597ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20062025-04-30T06:26:21.612ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20072025-04-30T06:26:21.612ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20082025-04-30T06:26:21.612ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20092025-04-30T06:26:21.612ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20102025-04-30T06:26:21.613ZDEBGtest_disk_expunge (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
20112025-04-30T06:26:21.613ZDEBGtest_disk_expunge (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 6799f3f8-4f25-496e-a002-28fa35199857 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
20122025-04-30T06:26:21.613ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
20132025-04-30T06:26:21.613ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20142025-04-30T06:26:21.613ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20152025-04-30T06:26:21.613ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20162025-04-30T06:26:21.615ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20172025-04-30T06:26:21.617ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
20182025-04-30T06:26:21.617ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20192025-04-30T06:26:21.617ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20202025-04-30T06:26:21.617ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20212025-04-30T06:26:21.618ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20222025-04-30T06:26:21.618ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20232025-04-30T06:26:21.619ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
20242025-04-30T06:26:21.619ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20252025-04-30T06:26:21.619ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a2 13 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58218 req_id = dab26c8c-f6c9-4d49-a96f-0dc724e8e95e
20262025-04-30T06:26:21.619ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41491,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58218 req_id = dab26c8c-f6c9-4d49-a96f-0dc724e8e95e
20272025-04-30T06:26:21.620ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
20282025-04-30T06:26:21.620ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41491, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58218 req_id = dab26c8c-f6c9-4d49-a96f-0dc724e8e95e
20292025-04-30T06:26:21.621ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
20302025-04-30T06:26:21.621ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
20312025-04-30T06:26:21.621ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
20322025-04-30T06:26:21.623ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20332025-04-30T06:26:21.654ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20342025-04-30T06:26:21.654ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20352025-04-30T06:26:21.654ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20362025-04-30T06:26:21.655ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20372025-04-30T06:26:21.658ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20382025-04-30T06:26:21.662ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20392025-04-30T06:26:21.662ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20402025-04-30T06:26:21.663ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20412025-04-30T06:26:21.663ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20422025-04-30T06:26:21.665ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e5f3ab6-87ef-44d5-8739-285d6c62d934", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
20432025-04-30T06:26:21.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
20442025-04-30T06:26:21.667ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bad14143-b199-4914-a11a-12e6ab432c9a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
20452025-04-30T06:26:21.667ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
20462025-04-30T06:26:21.725ZDEBGtest_disk_expunge (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 6799f3f8-4f25-496e-a002-28fa35199857 dns_zone_name = control-plane.oxide.internal found_so_far = 21 version = 1
20472025-04-30T06:26:21.726ZDEBGtest_disk_expunge (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20482025-04-30T06:26:21.727ZINFOtest_disk_expunge (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal generation = 1
20492025-04-30T06:26:21.727ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 327.268742ms iteration = 2 status = {"generation":1}
20502025-04-30T06:26:21.727ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Dependency
20512025-04-30T06:26:21.835ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[::1]:36493 generation = 1 method = PUT servers = DnsServersList { addresses: [[::1]:36493] } uri = http://[::1]:36493/config
20522025-04-30T06:26:21.836ZDEBGtest_disk_expunge (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20532025-04-30T06:26:21.836ZDEBGtest_disk_expunge (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 59b837ab-b516-4f5d-8b46-9770fb75a76f dns_zone_name = oxide-dev.test found_so_far = 0 version = 2
20542025-04-30T06:26:21.836ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
20552025-04-30T06:26:21.836ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20562025-04-30T06:26:21.836ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20572025-04-30T06:26:21.836ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20582025-04-30T06:26:21.837ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20592025-04-30T06:26:21.839ZINFOtest_disk_expunge (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-04-30T06:26:13.417360Z, time_modified: 2025-04-30T06:26:13.417360Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "test-suite-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 5c399561-4757-4f96-ac4f-8e23573dcd26, db_silo: Silo { identity: SiloIdentity { id: 5c399561-4757-4f96-ac4f-8e23573dcd26, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:26:20.588421Z, time_modified: 2025-04-30T06:26:20.588421Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide-dev.test" }, NoSiloCerts { silo_id: 5c399561-4757-4f96-ac4f-8e23573dcd26, dns_name: "test-suite-silo.sys.oxide-dev.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 5c399561-4757-4f96-ac4f-8e23573dcd26, db_silo: Silo { identity: SiloIdentity { id: 5c399561-4757-4f96-ac4f-8e23573dcd26, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:26:20.588421Z, time_modified: 2025-04-30T06:26:20.588421Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }) }
20602025-04-30T06:26:21.841ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 439.35754ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide-dev.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"test-suite-silo.sys.oxide-dev.test":{"silo_id":"5c399561-4757-4f96-ac4f-8e23573dcd26","tls_certs":[]}},"default_endpoint":{"silo_id":"5c399561-4757-4f96-ac4f-8e23573dcd26","tls_certs":[]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide-dev.test\\" has no usable certificates","silo 5c399561-4757-4f96-ac4f-8e23573dcd26 with DNS name \\"test-suite-silo.sys.oxide-dev.test\\" has no usable certificates"]}
20612025-04-30T06:26:21.847ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20622025-04-30T06:26:21.867ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20632025-04-30T06:26:21.867ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20642025-04-30T06:26:21.868ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20652025-04-30T06:26:21.868ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20662025-04-30T06:26:21.891ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
20672025-04-30T06:26:21.891ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
20682025-04-30T06:26:21.891ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20692025-04-30T06:26:21.891ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
20702025-04-30T06:26:21.892ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20712025-04-30T06:26:21.892ZINFOtest_disk_expunge (http): accepted connection kind = dns local_addr = [::1]:36493 remote_addr = [::1]:60579
20722025-04-30T06:26:21.894ZDEBGtest_disk_expunge (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 59b837ab-b516-4f5d-8b46-9770fb75a76f dns_zone_name = oxide-dev.test found_so_far = 1 version = 2
20732025-04-30T06:26:21.894ZDEBGtest_disk_expunge (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20742025-04-30T06:26:21.894ZINFOtest_disk_expunge (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external generation = 2
20752025-04-30T06:26:21.894ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 493.219397ms iteration = 2 status = {"generation":2}
20762025-04-30T06:26:21.894ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Dependency
20772025-04-30T06:26:21.964ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[::1]:52024 generation = 2 method = PUT servers = DnsServersList { addresses: [[::1]:52024] } uri = http://[::1]:52024/config
20782025-04-30T06:26:21.966ZINFOtest_disk_expunge (http): accepted connection kind = dns local_addr = [::1]:52024 remote_addr = [::1]:57466
20792025-04-30T06:26:21.967ZTRACtest_disk_expunge (http): incoming request kind = dns local_addr = [::1]:36493 method = PUT remote_addr = [::1]:60579 req_id = b1a5c6cb-e3c5-4c05-a8fe-f9f66c4b3aa8 uri = /config
20802025-04-30T06:26:21.968ZINFOtest_disk_expunge (store): attempting generation update new_generation = 1 req_id = b1a5c6cb-e3c5-4c05-a8fe-f9f66c4b3aa8
20812025-04-30T06:26:21.968ZINFOtest_disk_expunge (store): updated generation new_generation = 1 req_id = b1a5c6cb-e3c5-4c05-a8fe-f9f66c4b3aa8
20822025-04-30T06:26:21.968ZINFOtest_disk_expunge (http): request completed kind = dns latency_us = 1762 local_addr = [::1]:36493 method = PUT remote_addr = [::1]:60579 req_id = b1a5c6cb-e3c5-4c05-a8fe-f9f66c4b3aa8 response_code = 204 uri = /config
20832025-04-30T06:26:21.969ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
20842025-04-30T06:26:21.969ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20852025-04-30T06:26:21.969ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
20862025-04-30T06:26:21.970ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20872025-04-30T06:26:21.971ZTRACtest_disk_expunge (http): incoming request kind = dns local_addr = [::1]:52024 method = PUT remote_addr = [::1]:57466 req_id = 774c98c4-e269-4522-af0b-517b4b14d88e uri = /config
20882025-04-30T06:26:21.971ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:36493 generation = 1 result = Ok(Response { url: "http://[::1]:36493/config", status: 204, headers: {"x-request-id": "b1a5c6cb-e3c5-4c05-a8fe-f9f66c4b3aa8", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} }) servers = DnsServersList { addresses: [[::1]:36493] }
20892025-04-30T06:26:21.971ZINFOtest_disk_expunge (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:36493 generation = 1 servers = DnsServersList { addresses: [[::1]:36493] }
20902025-04-30T06:26:21.972ZINFOtest_disk_expunge (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal generation = 1 servers = DnsServersList { addresses: [[::1]:36493] }
20912025-04-30T06:26:21.973ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 245.43383ms iteration = 3 status = {"generation":1,"server_results":{"[::1]:36493":{"Ok":null}}}
20922025-04-30T06:26:21.973ZINFOtest_disk_expunge (store): attempting generation update new_generation = 2 req_id = 774c98c4-e269-4522-af0b-517b4b14d88e
20932025-04-30T06:26:21.973ZINFOtest_disk_expunge (store): pruning trees for generations newer than 0
20942025-04-30T06:26:21.973ZDEBGtest_disk_expunge (store): creating tree tree_name = generation_2_zone_oxide-dev.test
20952025-04-30T06:26:21.973ZDEBGtest_disk_expunge (store): updating current config
20962025-04-30T06:26:21.973ZDEBGtest_disk_expunge (store): flushing default tree
20972025-04-30T06:26:21.973ZINFOtest_disk_expunge (store): pruning trees for generations older than 2 keep = 3
20982025-04-30T06:26:21.973ZINFOtest_disk_expunge (store): updated generation new_generation = 2 req_id = 774c98c4-e269-4522-af0b-517b4b14d88e
20992025-04-30T06:26:21.975ZINFOtest_disk_expunge (http): request completed kind = dns latency_us = 4028 local_addr = [::1]:52024 method = PUT remote_addr = [::1]:57466 req_id = 774c98c4-e269-4522-af0b-517b4b14d88e response_code = 204 uri = /config
21002025-04-30T06:26:21.975ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:52024 generation = 2 result = Ok(Response { url: "http://[::1]:52024/config", status: 204, headers: {"x-request-id": "774c98c4-e269-4522-af0b-517b4b14d88e", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} }) servers = DnsServersList { addresses: [[::1]:52024] }
21012025-04-30T06:26:21.975ZINFOtest_disk_expunge (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:52024 generation = 2 servers = DnsServersList { addresses: [[::1]:52024] }
21022025-04-30T06:26:21.978ZINFOtest_disk_expunge (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external generation = 2 servers = DnsServersList { addresses: [[::1]:52024] }
21032025-04-30T06:26:21.978ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 84.532726ms iteration = 3 status = {"generation":2,"server_results":{"[::1]:52024":{"Ok":null}}}
21042025-04-30T06:26:21.980ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c96c436-033f-4602-9bb9-485a40c24b8e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
21052025-04-30T06:26:21.984ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
21062025-04-30T06:26:21.984ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21072025-04-30T06:26:21.984ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21082025-04-30T06:26:21.984ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21092025-04-30T06:26:21.985ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21102025-04-30T06:26:21.997ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b5d3bf0-1dfb-42af-b940-ec40334965eb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
21112025-04-30T06:26:21.998ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
21122025-04-30T06:26:21.998ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21132025-04-30T06:26:22.005ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21142025-04-30T06:26:22.005ZTRACtest_disk_expunge (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21152025-04-30T06:26:22.005ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21162025-04-30T06:26:22.005ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21172025-04-30T06:26:22.005ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21182025-04-30T06:26:22.005ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21192025-04-30T06:26:22.006ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21202025-04-30T06:26:22.008ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00d9e246-f8e9-4e08-bb19-d90593914070", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
21212025-04-30T06:26:22.008ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
21222025-04-30T06:26:22.008ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21232025-04-30T06:26:22.008ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21242025-04-30T06:26:22.008ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21252025-04-30T06:26:22.009ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21262025-04-30T06:26:22.016ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21272025-04-30T06:26:22.016ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21282025-04-30T06:26:22.016ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21292025-04-30T06:26:22.017ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21302025-04-30T06:26:22.024ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21312025-04-30T06:26:22.037ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21322025-04-30T06:26:22.037ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21332025-04-30T06:26:22.037ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21342025-04-30T06:26:22.038ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21352025-04-30T06:26:22.045ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21362025-04-30T06:26:22.045ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21372025-04-30T06:26:22.046ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21382025-04-30T06:26:22.051ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21392025-04-30T06:26:22.051ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21402025-04-30T06:26:22.051ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21412025-04-30T06:26:22.062ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "30b6bab9-ee56-4585-b9e4-96d6fee45286", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
21422025-04-30T06:26:22.062ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
21432025-04-30T06:26:22.062ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21442025-04-30T06:26:22.062ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21452025-04-30T06:26:22.063ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21462025-04-30T06:26:22.069ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "995f484e-a0d4-4290-b028-cb3fbc02ab04", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
21472025-04-30T06:26:22.070ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
21482025-04-30T06:26:22.070ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21492025-04-30T06:26:22.070ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21502025-04-30T06:26:22.070ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21512025-04-30T06:26:22.070ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21522025-04-30T06:26:22.071ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21532025-04-30T06:26:22.074ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21542025-04-30T06:26:22.074ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21552025-04-30T06:26:22.074ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21562025-04-30T06:26:22.074ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21572025-04-30T06:26:22.092ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "112b71ec-ba4b-43c1-a04a-a0b9fd82efae", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
21582025-04-30T06:26:22.092ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
21592025-04-30T06:26:22.092ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21602025-04-30T06:26:22.092ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21612025-04-30T06:26:22.092ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21622025-04-30T06:26:22.093ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21632025-04-30T06:26:22.095ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b0bbbb8-cfc5-47c0-abd1-e6833ee1c9fe", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
21642025-04-30T06:26:22.095ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
21652025-04-30T06:26:22.099ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21662025-04-30T06:26:22.107ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21672025-04-30T06:26:22.107ZTRACtest_disk_expunge (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21682025-04-30T06:26:22.107ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21692025-04-30T06:26:22.107ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21702025-04-30T06:26:22.107ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21712025-04-30T06:26:22.110ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f33c1a74-a392-435a-aea9-be2773614b67", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
21722025-04-30T06:26:22.110ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
21732025-04-30T06:26:22.112ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21742025-04-30T06:26:22.112ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21752025-04-30T06:26:22.120ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21762025-04-30T06:26:22.120ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8199262b-3664-464e-b459-ecb452b6cee9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
21772025-04-30T06:26:22.120ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
21782025-04-30T06:26:22.122ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c260c7f-55ef-4e9f-9c40-a1f1d4f7f02a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
21792025-04-30T06:26:22.122ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
21802025-04-30T06:26:22.122ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21812025-04-30T06:26:22.122ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21822025-04-30T06:26:22.122ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21832025-04-30T06:26:22.124ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f62a0c14-d5e3-402c-ab80-58af3f5b161e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
21842025-04-30T06:26:22.124ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
21852025-04-30T06:26:22.125ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21862025-04-30T06:26:22.125ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21872025-04-30T06:26:22.125ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21882025-04-30T06:26:22.128ZTRACtest_disk_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
21892025-04-30T06:26:22.128ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21902025-04-30T06:26:22.128ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21912025-04-30T06:26:22.128ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21922025-04-30T06:26:22.128ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21932025-04-30T06:26:22.129ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a54a3cb5-171e-4aea-8f58-f30df9cd7952", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
21942025-04-30T06:26:22.129ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
21952025-04-30T06:26:22.130ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21962025-04-30T06:26:22.130ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21972025-04-30T06:26:22.130ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
21982025-04-30T06:26:22.130ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21992025-04-30T06:26:22.132ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b85266e9-b963-4f11-b461-5413a1a03ce7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22002025-04-30T06:26:22.132ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
22012025-04-30T06:26:22.133ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22022025-04-30T06:26:22.133ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22032025-04-30T06:26:22.133ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22042025-04-30T06:26:22.133ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22052025-04-30T06:26:22.135ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "732e95e3-dd8f-41ac-a0bc-a2100e3fa1a6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22062025-04-30T06:26:22.135ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
22072025-04-30T06:26:22.135ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22082025-04-30T06:26:22.156ZDEBGtest_disk_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22092025-04-30T06:26:22.156ZTRACtest_disk_expunge (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22102025-04-30T06:26:22.156ZTRACtest_disk_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22112025-04-30T06:26:22.156ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22122025-04-30T06:26:22.156ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22132025-04-30T06:26:22.156ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22142025-04-30T06:26:22.157ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22152025-04-30T06:26:22.160ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22162025-04-30T06:26:22.160ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22172025-04-30T06:26:22.161ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22182025-04-30T06:26:22.162ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22192025-04-30T06:26:22.165ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22202025-04-30T06:26:22.165ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22212025-04-30T06:26:22.165ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22222025-04-30T06:26:22.166ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22232025-04-30T06:26:22.168ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dd407f5-4216-4b1c-a4b4-ac579ef2c91a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22242025-04-30T06:26:22.168ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
22252025-04-30T06:26:22.169ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22262025-04-30T06:26:22.176ZDEBGtest_disk_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22272025-04-30T06:26:22.176ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22282025-04-30T06:26:22.176ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22292025-04-30T06:26:22.177ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22302025-04-30T06:26:22.185ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b90251d-a26a-4a98-81f4-5ea72d3df76b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22312025-04-30T06:26:22.185ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
22322025-04-30T06:26:22.186ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22332025-04-30T06:26:22.186ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22342025-04-30T06:26:22.187ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22352025-04-30T06:26:22.191ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22362025-04-30T06:26:22.191ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22372025-04-30T06:26:22.192ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22382025-04-30T06:26:22.194ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d1561e3-5f65-493a-be66-3915c486ebb6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22392025-04-30T06:26:22.194ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
22402025-04-30T06:26:22.195ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22412025-04-30T06:26:22.195ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22422025-04-30T06:26:22.195ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22432025-04-30T06:26:22.197ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0733bba-ae83-452a-98df-c00ca0e095fd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22442025-04-30T06:26:22.197ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
22452025-04-30T06:26:22.201ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22462025-04-30T06:26:22.201ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22472025-04-30T06:26:22.202ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22482025-04-30T06:26:22.203ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13cf1eb7-ec08-4442-9c9a-a2e78baff5f3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22492025-04-30T06:26:22.203ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
22502025-04-30T06:26:22.205ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7451ae1-6dbc-45a1-a6db-88e21acee7fd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22512025-04-30T06:26:22.205ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
22522025-04-30T06:26:22.206ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd639a68-4ae1-4f4d-a9e0-0288000800fd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22532025-04-30T06:26:22.206ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
22542025-04-30T06:26:22.207ZTRACtest_disk_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22552025-04-30T06:26:22.207ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22562025-04-30T06:26:22.207ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22572025-04-30T06:26:22.207ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22582025-04-30T06:26:22.208ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22592025-04-30T06:26:22.208ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a11a6061-e3e2-4883-b17a-56fade597bba", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22602025-04-30T06:26:22.208ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
22612025-04-30T06:26:22.211ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22622025-04-30T06:26:22.211ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22632025-04-30T06:26:22.211ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22642025-04-30T06:26:22.211ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22652025-04-30T06:26:22.213ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22662025-04-30T06:26:22.213ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22672025-04-30T06:26:22.213ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22682025-04-30T06:26:22.214ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22692025-04-30T06:26:22.229ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22702025-04-30T06:26:22.240ZDEBGtest_disk_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22712025-04-30T06:26:22.240ZTRACtest_disk_expunge (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22722025-04-30T06:26:22.240ZTRACtest_disk_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22732025-04-30T06:26:22.240ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22742025-04-30T06:26:22.240ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22752025-04-30T06:26:22.240ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22762025-04-30T06:26:22.241ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22772025-04-30T06:26:22.241ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7d96c85-8f7d-415a-a163-262713cabad5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22782025-04-30T06:26:22.241ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
22792025-04-30T06:26:22.246ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22802025-04-30T06:26:22.246ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22812025-04-30T06:26:22.246ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22822025-04-30T06:26:22.247ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22832025-04-30T06:26:22.248ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24a1ffce-bc1d-4f47-856e-24d9b0b5839b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22842025-04-30T06:26:22.248ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
22852025-04-30T06:26:22.252ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceafe3de-0cfe-4cf9-bad9-f5f4192a9aad", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22862025-04-30T06:26:22.252ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
22872025-04-30T06:26:22.253ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22882025-04-30T06:26:22.253ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22892025-04-30T06:26:22.253ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22902025-04-30T06:26:22.254ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22912025-04-30T06:26:22.274ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22922025-04-30T06:26:22.287ZDEBGtest_disk_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22932025-04-30T06:26:22.287ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22942025-04-30T06:26:22.287ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
22952025-04-30T06:26:22.302ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22962025-04-30T06:26:22.305ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00ca7ffd-c1cc-45cb-906a-2e95d317864d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22972025-04-30T06:26:22.305ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
22982025-04-30T06:26:22.311ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bd14daa-2f6d-4c1a-98c1-f17e4548570c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
22992025-04-30T06:26:22.311ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
23002025-04-30T06:26:22.315ZDEBGtest_disk_expunge (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 76b6444e-557a-4565-9b1f-2d99179e79ac, kind: Service { id: 78121327-166d-4e33-935c-82ab8635bf33 }, name: Name("nexus-78121327-166d-4e33-935c-82ab8635bf33"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: dbdfd5b5-8b4b-474d-98e3-880e650dc7ed, kind: Service { id: 68c89553-4ad9-4464-83e9-6636df41c29b }, name: Name("external-dns-68c89553-4ad9-4464-83e9-6636df41c29b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
23012025-04-30T06:26:22.317ZTRACtest_disk_expunge (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
23022025-04-30T06:26:22.318ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23032025-04-30T06:26:22.336ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23042025-04-30T06:26:22.336ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
23052025-04-30T06:26:22.337ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23062025-04-30T06:26:22.340ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23072025-04-30T06:26:22.363ZDEBGtest_disk_expunge (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
23082025-04-30T06:26:22.363ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23092025-04-30T06:26:22.363ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
23102025-04-30T06:26:22.364ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23112025-04-30T06:26:22.365ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fb87f91-7d8a-49bd-bba6-38d4f79e4066", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
23122025-04-30T06:26:22.366ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
23132025-04-30T06:26:22.366ZTRACtest_disk_expunge (InternalApi): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:26:20.791586Z, allowed_ips: None }
23142025-04-30T06:26:22.366ZDEBGtest_disk_expunge (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
23152025-04-30T06:26:22.367ZDEBGtest_disk_expunge (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: dbdfd5b5-8b4b-474d-98e3-880e650dc7ed, kind: Service { id: 68c89553-4ad9-4464-83e9-6636df41c29b }, name: Name("external-dns-68c89553-4ad9-4464-83e9-6636df41c29b"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 76b6444e-557a-4565-9b1f-2d99179e79ac, kind: Service { id: 78121327-166d-4e33-935c-82ab8635bf33 }, name: Name("nexus-78121327-166d-4e33-935c-82ab8635bf33"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
23162025-04-30T06:26:22.367ZDEBGtest_disk_expunge (InternalApi): resolved 2 rules for sleds
23172025-04-30T06:26:22.367ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35913f4b-6a56-458e-a85c-3ba4b408a9d9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
23182025-04-30T06:26:22.367ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
23192025-04-30T06:26:22.368ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d97c05d1-700c-4d2b-aa4e-188bbfe39587", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
23202025-04-30T06:26:22.369ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
23212025-04-30T06:26:22.370ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f8d8f5d-eb77-4a6e-ab8f-9199561ff0be", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
23222025-04-30T06:26:22.370ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
23232025-04-30T06:26:22.371ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19b10c9e-4e16-49c3-b2d4-7bb7a9e1da07", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
23242025-04-30T06:26:22.371ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
23252025-04-30T06:26:22.373ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a633a809-33b2-4397-b56b-f790682a8cb3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
23262025-04-30T06:26:22.373ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
23272025-04-30T06:26:22.376ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb1d6254-f1b6-419c-9e99-e6bb2161f0e9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
23282025-04-30T06:26:22.376ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
23292025-04-30T06:26:22.378ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7355d2bd-f87d-4e04-b070-6481208fb51e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
23302025-04-30T06:26:22.378ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
23312025-04-30T06:26:22.380ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f97c9965-7576-4907-94e9-e8a462a61060", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
23322025-04-30T06:26:22.380ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
23332025-04-30T06:26:22.382ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13bd3333-bc15-4670-91e3-9a5e36d2a49e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
23342025-04-30T06:26:22.382ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
23352025-04-30T06:26:22.384ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "129ac32c-ea9b-405c-b5f2-94a92b555dba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
23362025-04-30T06:26:22.384ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
23372025-04-30T06:26:22.390ZDEBGtest_disk_expunge (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:26:14.842443Z, time_modified: 2025-04-30T06:26:14.842443Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(44559), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(47123) }]
23382025-04-30T06:26:22.390ZDEBGtest_disk_expunge (InternalApi): sending firewall rules to sled agents
23392025-04-30T06:26:22.390ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23402025-04-30T06:26:22.390ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
23412025-04-30T06:26:22.391ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23422025-04-30T06:26:22.393ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f955594c-a805-4345-9ebc-15b074899bab", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:21 GMT"} })
23432025-04-30T06:26:22.393ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
23442025-04-30T06:26:22.394ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
23452025-04-30T06:26:22.394ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23462025-04-30T06:26:22.394ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23472025-04-30T06:26:22.394ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
23482025-04-30T06:26:22.394ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23492025-04-30T06:26:22.398ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23502025-04-30T06:26:22.406ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
23512025-04-30T06:26:22.545ZDEBGtest_disk_expunge (InternalApi): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:44559/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23522025-04-30T06:26:22.545ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4319ee78-347d-42b8-8875-b44160768a7f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:22 GMT"} })
23532025-04-30T06:26:22.546ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
23542025-04-30T06:26:22.546ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:53765 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23552025-04-30T06:26:22.548ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = PUT remote_addr = [::1]:53765 req_id = c4495c8f-2e45-4031-8f98-fc67fd38ec4d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23562025-04-30T06:26:22.550ZINFOtest_disk_expunge (dropshot): request completed latency_us = 2057 local_addr = [::1]:44559 method = PUT remote_addr = [::1]:53765 req_id = c4495c8f-2e45-4031-8f98-fc67fd38ec4d response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23572025-04-30T06:26:22.550ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce1971c5-8e27-4e4a-80a1-2fd14a432246", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:22 GMT"} })
23582025-04-30T06:26:22.550ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
23592025-04-30T06:26:22.551ZDEBGtest_disk_expunge (InternalApi): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:44559/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "c4495c8f-2e45-4031-8f98-fc67fd38ec4d", "date": "Wed, 30 Apr 2025 06:26:22 GMT"} })
23602025-04-30T06:26:22.554ZINFOtest_disk_expunge (InternalApi): updated firewall rules on 1 sleds
23612025-04-30T06:26:22.554ZINFOtest_disk_expunge (nexus): plumbed initial IP allowlist
23622025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = / versions = all
23632025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /access versions = all
23642025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /assets/{path} versions = all
23652025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /device/auth versions = all
23662025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /device/confirm versions = all
23672025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /device/success versions = all
23682025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /device/token versions = all
23692025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /device/verify versions = all
23702025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /experimental/v1/probes versions = all
23712025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /experimental/v1/probes versions = all
23722025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /experimental/v1/probes/{probe} versions = all
23732025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /experimental/v1/probes/{probe} versions = all
23742025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /experimental/v1/system/support-bundles versions = all
23752025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /experimental/v1/system/support-bundles versions = all
23762025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23772025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
23782025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23792025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
23802025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
23812025-04-30T06:26:22.655ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
23822025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
23832025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /images versions = all
23842025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /login versions = all
23852025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /login/{silo_name}/local versions = all
23862025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
23872025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
23882025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
23892025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /lookup/{path} versions = all
23902025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /projects/{path} versions = all
23912025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /projects-new versions = all
23922025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /settings/{path} versions = all
23932025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /system/{path} versions = all
23942025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /utilization versions = all
23952025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/affinity-groups versions = all
23962025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/affinity-groups versions = all
23972025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
23982025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
23992025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
24002025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
24012025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24022025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24032025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24042025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/anti-affinity-groups versions = all
24052025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/anti-affinity-groups versions = all
24062025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24072025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24082025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24092025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
24102025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24112025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24122025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24132025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/certificates versions = all
24142025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/certificates versions = all
24152025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/certificates/{certificate} versions = all
24162025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/certificates/{certificate} versions = all
24172025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/disks versions = all
24182025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/disks versions = all
24192025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/disks/{disk} versions = all
24202025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/disks/{disk} versions = all
24212025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/disks/{disk}/bulk-write versions = all
24222025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
24232025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
24242025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/disks/{disk}/finalize versions = all
24252025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
24262025-04-30T06:26:22.656ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/floating-ips versions = all
24272025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/floating-ips versions = all
24282025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
24292025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/floating-ips/{floating_ip} versions = all
24302025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
24312025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
24322025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
24332025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/groups versions = all
24342025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/groups/{group_id} versions = all
24352025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/images versions = all
24362025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/images versions = all
24372025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/images/{image} versions = all
24382025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/images/{image} versions = all
24392025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/images/{image}/demote versions = all
24402025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/images/{image}/promote versions = all
24412025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/instances versions = all
24422025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/instances versions = all
24432025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/instances/{instance} versions = all
24442025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/instances/{instance} versions = all
24452025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/instances/{instance} versions = all
24462025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
24472025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
24482025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/instances/{instance}/disks versions = all
24492025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/instances/{instance}/disks/attach versions = all
24502025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/instances/{instance}/disks/detach versions = all
24512025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/instances/{instance}/external-ips versions = all
24522025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24532025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24542025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/instances/{instance}/reboot versions = all
24552025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/instances/{instance}/serial-console versions = all
24562025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
24572025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
24582025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/instances/{instance}/start versions = all
24592025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/instances/{instance}/stop versions = all
24602025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/internet-gateway-ip-addresses versions = all
24612025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/internet-gateway-ip-addresses versions = all
24622025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
24632025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/internet-gateway-ip-pools versions = all
24642025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/internet-gateway-ip-pools versions = all
24652025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
24662025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/internet-gateways versions = all
24672025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/internet-gateways versions = all
24682025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
24692025-04-30T06:26:22.657ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/internet-gateways/{gateway} versions = all
24702025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/ip-pools versions = all
24712025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/ip-pools/{pool} versions = all
24722025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/login/{silo_name}/local versions = all
24732025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/logout versions = all
24742025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/me versions = all
24752025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/me/groups versions = all
24762025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/me/ssh-keys versions = all
24772025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/me/ssh-keys versions = all
24782025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
24792025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
24802025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/metrics/{metric_name} versions = all
24812025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/network-interfaces versions = all
24822025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/network-interfaces versions = all
24832025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/network-interfaces/{interface} versions = all
24842025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/network-interfaces/{interface} versions = all
24852025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/network-interfaces/{interface} versions = all
24862025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/ping versions = all
24872025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/policy versions = all
24882025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/policy versions = all
24892025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/projects versions = all
24902025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/projects versions = all
24912025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/projects/{project} versions = all
24922025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/projects/{project} versions = all
24932025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/projects/{project} versions = all
24942025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/projects/{project}/policy versions = all
24952025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/projects/{project}/policy versions = all
24962025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/snapshots versions = all
24972025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/snapshots versions = all
24982025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/snapshots/{snapshot} versions = all
24992025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/snapshots/{snapshot} versions = all
25002025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/disks versions = all
25012025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
25022025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
25032025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/racks versions = all
25042025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
25052025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/sleds versions = all
25062025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/hardware/sleds versions = all
25072025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
25082025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
25092025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
25102025-04-30T06:26:22.658ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
25112025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
25122025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/switch-port versions = all
25132025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25142025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25152025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
25162025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
25172025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
25182025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/switches versions = all
25192025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
25202025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/identity-providers versions = all
25212025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/identity-providers/local/users versions = all
25222025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
25232025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
25242025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/identity-providers/saml versions = all
25252025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
25262025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/ip-pools versions = all
25272025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/ip-pools versions = all
25282025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
25292025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/ip-pools/{pool} versions = all
25302025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/system/ip-pools/{pool} versions = all
25312025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
25322025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
25332025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
25342025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
25352025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
25362025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25372025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25382025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
25392025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/ip-pools-service versions = all
25402025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/ip-pools-service/ranges versions = all
25412025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
25422025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
25432025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/metrics/{metric_name} versions = all
25442025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/address-lot versions = all
25452025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/networking/address-lot versions = all
25462025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
25472025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
25482025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/allow-list versions = all
25492025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/system/networking/allow-list versions = all
25502025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/networking/bfd-disable versions = all
25512025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/networking/bfd-enable versions = all
25522025-04-30T06:26:22.659ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/bfd-status versions = all
25532025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/system/networking/bgp versions = all
25542025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/bgp versions = all
25552025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/networking/bgp versions = all
25562025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/bgp-announce-set versions = all
25572025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
25582025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
25592025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
25602025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/bgp-exported versions = all
25612025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/bgp-message-history versions = all
25622025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
25632025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/bgp-status versions = all
25642025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/loopback-address versions = all
25652025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/networking/loopback-address versions = all
25662025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
25672025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
25682025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/switch-port-settings versions = all
25692025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/networking/switch-port-settings versions = all
25702025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
25712025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/policy versions = all
25722025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/system/policy versions = all
25732025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/roles versions = all
25742025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/roles/{role_name} versions = all
25752025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/silo-quotas versions = all
25762025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/silos versions = all
25772025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/silos versions = all
25782025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/system/silos/{silo} versions = all
25792025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/silos/{silo} versions = all
25802025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
25812025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/silos/{silo}/policy versions = all
25822025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/system/silos/{silo}/policy versions = all
25832025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/silos/{silo}/quotas versions = all
25842025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
25852025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/system/timeseries/query versions = all
25862025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/timeseries/schemas versions = all
25872025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/system/update/repository versions = all
25882025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/update/repository/{system_version} versions = all
25892025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/update/target-release versions = all
25902025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/system/update/target-release versions = all
25912025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/users versions = all
25922025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/users/{user_id} versions = all
25932025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/users-builtin versions = all
25942025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/users-builtin/{user} versions = all
25952025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/utilization/silos versions = all
25962025-04-30T06:26:22.660ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/system/utilization/silos/{silo} versions = all
25972025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/timeseries/query versions = all
25982025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/users versions = all
25992025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/utilization versions = all
26002025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/vpc-firewall-rules versions = all
26012025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/vpc-firewall-rules versions = all
26022025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/vpc-router-routes versions = all
26032025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/vpc-router-routes versions = all
26042025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
26052025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/vpc-router-routes/{route} versions = all
26062025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/vpc-router-routes/{route} versions = all
26072025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/vpc-routers versions = all
26082025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/vpc-routers versions = all
26092025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/vpc-routers/{router} versions = all
26102025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/vpc-routers/{router} versions = all
26112025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/vpc-routers/{router} versions = all
26122025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/vpc-subnets versions = all
26132025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/vpc-subnets versions = all
26142025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
26152025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/vpc-subnets/{subnet} versions = all
26162025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
26172025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
26182025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/vpcs versions = all
26192025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/vpcs versions = all
26202025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/vpcs/{vpc} versions = all
26212025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/vpcs/{vpc} versions = all
26222025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/vpcs/{vpc} versions = all
26232025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/webhooks/deliveries versions = all
26242025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
26252025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/webhooks/event-classes versions = all
26262025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/webhooks/receivers versions = all
26272025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/webhooks/receivers versions = all
26282025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
26292025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
26302025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
26312025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
26322025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
26332025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
26342025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = GET path = /v1/webhooks/secrets versions = all
26352025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = POST path = /v1/webhooks/secrets versions = all
26362025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): registered endpoint local_addr = 127.0.0.1:57204 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
26372025-04-30T06:26:22.661ZINFOtest_disk_expunge (dropshot_external): listening local_addr = 127.0.0.1:57204
26382025-04-30T06:26:22.661ZDEBGtest_disk_expunge (dropshot_external): successfully registered DTrace USDT probes local_addr = 127.0.0.1:57204
26392025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = / versions = all
26402025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /access versions = all
26412025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /assets/{path} versions = all
26422025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /device/auth versions = all
26432025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /device/confirm versions = all
26442025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /device/success versions = all
26452025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /device/token versions = all
26462025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /device/verify versions = all
26472025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /experimental/v1/probes versions = all
26482025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /experimental/v1/probes versions = all
26492025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /experimental/v1/probes/{probe} versions = all
26502025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /experimental/v1/probes/{probe} versions = all
26512025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /experimental/v1/system/support-bundles versions = all
26522025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /experimental/v1/system/support-bundles versions = all
26532025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26542025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26552025-04-30T06:26:22.715ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26562025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26572025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26582025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26592025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
26602025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /images versions = all
26612025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /login versions = all
26622025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /login/{silo_name}/local versions = all
26632025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
26642025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
26652025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
26662025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /lookup/{path} versions = all
26672025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /projects/{path} versions = all
26682025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /projects-new versions = all
26692025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /settings/{path} versions = all
26702025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /system/{path} versions = all
26712025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /utilization versions = all
26722025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/affinity-groups versions = all
26732025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/affinity-groups versions = all
26742025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
26752025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
26762025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
26772025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
26782025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26792025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26802025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
26812025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/anti-affinity-groups versions = all
26822025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/anti-affinity-groups versions = all
26832025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26842025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26852025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
26862025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
26872025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26882025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26892025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
26902025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/certificates versions = all
26912025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/certificates versions = all
26922025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/certificates/{certificate} versions = all
26932025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/certificates/{certificate} versions = all
26942025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/disks versions = all
26952025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/disks versions = all
26962025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/disks/{disk} versions = all
26972025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/disks/{disk} versions = all
26982025-04-30T06:26:22.716ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/disks/{disk}/bulk-write versions = all
26992025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
27002025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
27012025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/disks/{disk}/finalize versions = all
27022025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
27032025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/floating-ips versions = all
27042025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/floating-ips versions = all
27052025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
27062025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/floating-ips/{floating_ip} versions = all
27072025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
27082025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
27092025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
27102025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/groups versions = all
27112025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/groups/{group_id} versions = all
27122025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/images versions = all
27132025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/images versions = all
27142025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/images/{image} versions = all
27152025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/images/{image} versions = all
27162025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/images/{image}/demote versions = all
27172025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/images/{image}/promote versions = all
27182025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/instances versions = all
27192025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/instances versions = all
27202025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/instances/{instance} versions = all
27212025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/instances/{instance} versions = all
27222025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/instances/{instance} versions = all
27232025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
27242025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
27252025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/instances/{instance}/disks versions = all
27262025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/instances/{instance}/disks/attach versions = all
27272025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/instances/{instance}/disks/detach versions = all
27282025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/instances/{instance}/external-ips versions = all
27292025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27302025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27312025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/instances/{instance}/reboot versions = all
27322025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/instances/{instance}/serial-console versions = all
27332025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
27342025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
27352025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/instances/{instance}/start versions = all
27362025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/instances/{instance}/stop versions = all
27372025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/internet-gateway-ip-addresses versions = all
27382025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/internet-gateway-ip-addresses versions = all
27392025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
27402025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/internet-gateway-ip-pools versions = all
27412025-04-30T06:26:22.717ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/internet-gateway-ip-pools versions = all
27422025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
27432025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/internet-gateways versions = all
27442025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/internet-gateways versions = all
27452025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
27462025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/internet-gateways/{gateway} versions = all
27472025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/ip-pools versions = all
27482025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/ip-pools/{pool} versions = all
27492025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/login/{silo_name}/local versions = all
27502025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/logout versions = all
27512025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/me versions = all
27522025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/me/groups versions = all
27532025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/me/ssh-keys versions = all
27542025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/me/ssh-keys versions = all
27552025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
27562025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
27572025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/metrics/{metric_name} versions = all
27582025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/network-interfaces versions = all
27592025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/network-interfaces versions = all
27602025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/network-interfaces/{interface} versions = all
27612025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/network-interfaces/{interface} versions = all
27622025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/network-interfaces/{interface} versions = all
27632025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/ping versions = all
27642025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/policy versions = all
27652025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/policy versions = all
27662025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/projects versions = all
27672025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/projects versions = all
27682025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/projects/{project} versions = all
27692025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/projects/{project} versions = all
27702025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/projects/{project} versions = all
27712025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/projects/{project}/policy versions = all
27722025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/projects/{project}/policy versions = all
27732025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/snapshots versions = all
27742025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/snapshots versions = all
27752025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/snapshots/{snapshot} versions = all
27762025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/snapshots/{snapshot} versions = all
27772025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/disks versions = all
27782025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
27792025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
27802025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/racks versions = all
27812025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
27822025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/sleds versions = all
27832025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/hardware/sleds versions = all
27842025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
27852025-04-30T06:26:22.718ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
27862025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
27872025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
27882025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
27892025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/switch-port versions = all
27902025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27912025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
27922025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
27932025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
27942025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
27952025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/switches versions = all
27962025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
27972025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/identity-providers versions = all
27982025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/identity-providers/local/users versions = all
27992025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
28002025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
28012025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/identity-providers/saml versions = all
28022025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
28032025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/ip-pools versions = all
28042025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/ip-pools versions = all
28052025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
28062025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/ip-pools/{pool} versions = all
28072025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/system/ip-pools/{pool} versions = all
28082025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
28092025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
28102025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
28112025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
28122025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
28132025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28142025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28152025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
28162025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/ip-pools-service versions = all
28172025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/ip-pools-service/ranges versions = all
28182025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
28192025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
28202025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/metrics/{metric_name} versions = all
28212025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/address-lot versions = all
28222025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/networking/address-lot versions = all
28232025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
28242025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
28252025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/allow-list versions = all
28262025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/system/networking/allow-list versions = all
28272025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/networking/bfd-disable versions = all
28282025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/networking/bfd-enable versions = all
28292025-04-30T06:26:22.719ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/bfd-status versions = all
28302025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/system/networking/bgp versions = all
28312025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/bgp versions = all
28322025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/networking/bgp versions = all
28332025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/bgp-announce-set versions = all
28342025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
28352025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
28362025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
28372025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/bgp-exported versions = all
28382025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/bgp-message-history versions = all
28392025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
28402025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/bgp-status versions = all
28412025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/loopback-address versions = all
28422025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/networking/loopback-address versions = all
28432025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
28442025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
28452025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/switch-port-settings versions = all
28462025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/networking/switch-port-settings versions = all
28472025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
28482025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/policy versions = all
28492025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/system/policy versions = all
28502025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/roles versions = all
28512025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/roles/{role_name} versions = all
28522025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/silo-quotas versions = all
28532025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/silos versions = all
28542025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/silos versions = all
28552025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/system/silos/{silo} versions = all
28562025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/silos/{silo} versions = all
28572025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
28582025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/silos/{silo}/policy versions = all
28592025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/system/silos/{silo}/policy versions = all
28602025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/silos/{silo}/quotas versions = all
28612025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
28622025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/system/timeseries/query versions = all
28632025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/timeseries/schemas versions = all
28642025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/system/update/repository versions = all
28652025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/update/repository/{system_version} versions = all
28662025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/update/target-release versions = all
28672025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/system/update/target-release versions = all
28682025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/users versions = all
28692025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/users/{user_id} versions = all
28702025-04-30T06:26:22.720ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/users-builtin versions = all
28712025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/users-builtin/{user} versions = all
28722025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/utilization/silos versions = all
28732025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/system/utilization/silos/{silo} versions = all
28742025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/timeseries/query versions = all
28752025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/users versions = all
28762025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/utilization versions = all
28772025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/vpc-firewall-rules versions = all
28782025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/vpc-firewall-rules versions = all
28792025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/vpc-router-routes versions = all
28802025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/vpc-router-routes versions = all
28812025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
28822025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/vpc-router-routes/{route} versions = all
28832025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/vpc-router-routes/{route} versions = all
28842025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/vpc-routers versions = all
28852025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/vpc-routers versions = all
28862025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/vpc-routers/{router} versions = all
28872025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/vpc-routers/{router} versions = all
28882025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/vpc-routers/{router} versions = all
28892025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/vpc-subnets versions = all
28902025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/vpc-subnets versions = all
28912025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
28922025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/vpc-subnets/{subnet} versions = all
28932025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
28942025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
28952025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/vpcs versions = all
28962025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/vpcs versions = all
28972025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/vpcs/{vpc} versions = all
28982025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/vpcs/{vpc} versions = all
28992025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/vpcs/{vpc} versions = all
29002025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/webhooks/deliveries versions = all
29012025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
29022025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/webhooks/event-classes versions = all
29032025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/webhooks/receivers versions = all
29042025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/webhooks/receivers versions = all
29052025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
29062025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
29072025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
29082025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
29092025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
29102025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
29112025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = GET path = /v1/webhooks/secrets versions = all
29122025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = POST path = /v1/webhooks/secrets versions = all
29132025-04-30T06:26:22.721ZDEBGtest_disk_expunge (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:37940 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
29142025-04-30T06:26:22.721ZINFOtest_disk_expunge (dropshot_external_techport): listening local_addr = 127.0.0.1:37940
29152025-04-30T06:26:22.722ZDEBGtest_disk_expunge (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = 127.0.0.1:37940
29162025-04-30T06:26:22.722ZDEBGtest_disk_expunge (nexus-producer-server): registered DTrace probes
29172025-04-30T06:26:22.722ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = 127.0.0.1:38728 method = GET path = /{producer_id} versions = all
29182025-04-30T06:26:22.722ZINFOtest_disk_expunge (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 127.0.0.1:38728
29192025-04-30T06:26:22.722ZDEBGtest_disk_expunge (dropshot): successfully registered DTrace USDT probes local_addr = 127.0.0.1:38728
29202025-04-30T06:26:22.722ZDEBGtest_disk_expunge (nexus-producer-server): Requested any available port, Dropshot server has been bound to 127.0.0.1:38728
29212025-04-30T06:26:22.722ZDEBGtest_disk_expunge (nexus-producer-server): Nexus IP provided explicitly, will use it addr = 127.0.0.1:34036
29222025-04-30T06:26:22.722ZDEBGtest_disk_expunge (nexus-producer-server): starting producer registration task
29232025-04-30T06:26:22.722ZINFOtest_disk_expunge (nexus-producer-server): starting oximeter metric producer server address = 127.0.0.1:38728 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = 78121327-166d-4e33-935c-82ab8635bf33
29242025-04-30T06:26:22.722ZDEBGtest_disk_expunge (init_with_steps): Running step start_oximeter
29252025-04-30T06:26:22.722ZDEBGtest_disk_expunge: Starting Oximeter
29262025-04-30T06:26:22.722ZDEBGtest_disk_expunge (oximeter): registered DTrace probes
29272025-04-30T06:26:22.722ZINFOtest_disk_expunge (oximeter): starting oximeter server file = oximeter/collector/src/lib.rs:236
29282025-04-30T06:26:22.722ZDEBGtest_disk_expunge (oximeter): creating ClickHouse client
29292025-04-30T06:26:22.722ZDEBGtest_disk_expunge (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
29302025-04-30T06:26:22.722ZDEBGtest_disk_expunge (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0
29312025-04-30T06:26:22.723ZDEBGtest_disk_expunge (producer-registration-task): will register with Nexus at 127.0.0.1:34036
29322025-04-30T06:26:22.769ZDEBGtest_disk_expunge (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:34036/metrics/producers
29332025-04-30T06:26:22.771ZINFOtest_disk_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:34036 remote_addr = 127.0.0.1:49969
29342025-04-30T06:26:22.773ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:49969 req_id = dc879631-7f6b-4ce9-9913-0934840295cb uri = /metrics/producers
29352025-04-30T06:26:22.775ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:49969 req_id = dc879631-7f6b-4ce9-9913-0934840295cb resource = Database uri = /metrics/producers
29362025-04-30T06:26:22.775ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:49969 req_id = dc879631-7f6b-4ce9-9913-0934840295cb roles = RoleSet { roles: {} } uri = /metrics/producers
29372025-04-30T06:26:22.776ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:49969 req_id = dc879631-7f6b-4ce9-9913-0934840295cb resource = Database result = Ok(()) uri = /metrics/producers
29382025-04-30T06:26:22.783ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29392025-04-30T06:26:22.794ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81140eea-2137-4aed-b0ca-a19aa7c48848", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:22 GMT"} })
29402025-04-30T06:26:22.795ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
29412025-04-30T06:26:22.800ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e15cb81-e254-445d-b69a-a9740c331f0a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:22 GMT"} })
29422025-04-30T06:26:22.800ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29432025-04-30T06:26:22.801ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29442025-04-30T06:26:22.801ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29452025-04-30T06:26:22.808ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29462025-04-30T06:26:22.808ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29472025-04-30T06:26:22.809ZWARNtest_disk_expunge (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
29482025-04-30T06:26:22.809ZDEBGtest_disk_expunge (oximeter-agent): oximeter database does not exist, creating collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
29492025-04-30T06:26:22.809ZINFOtest_disk_expunge (clickhouse-client): reading db version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:794 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
29502025-04-30T06:26:22.809ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29512025-04-30T06:26:22.813ZWARNtest_disk_expunge (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
29522025-04-30T06:26:22.814ZINFOtest_disk_expunge (clickhouse-client): read oximeter database version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:799 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 version = 0
29532025-04-30T06:26:22.814ZINFOtest_disk_expunge (clickhouse-client): initializing oximeter schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:804 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
29542025-04-30T06:26:22.814ZDEBGtest_disk_expunge (clickhouse-client): initializing ClickHouse database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
29552025-04-30T06:26:22.814ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29562025-04-30T06:26:22.824ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
29572025-04-30T06:26:22.824ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
29582025-04-30T06:26:22.824ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29592025-04-30T06:26:22.824ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29602025-04-30T06:26:22.824ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29612025-04-30T06:26:22.825ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29622025-04-30T06:26:22.826ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = CREATE DATABASE IF NOT EXISTS oximeter
29632025-04-30T06:26:22.829ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\n/* The version table contains metadata about the `oximeter` database */\nCREATE TABLE IF NOT EXISTS oximeter.version\n(\n value UInt64,\n timestamp DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (value, timestamp)
29642025-04-30T06:26:22.830ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29652025-04-30T06:26:22.834ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
29662025-04-30T06:26:22.834ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29672025-04-30T06:26:22.834ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29682025-04-30T06:26:22.835ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29692025-04-30T06:26:22.835ZINFOtest_disk_expunge (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 62667 local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:49969 req_id = dc879631-7f6b-4ce9-9913-0934840295cb response_code = 503 uri = /metrics/producers
29702025-04-30T06:26:22.835ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\n/* The measurement tables contain all individual samples from each timeseries.\n *\n * Each table stores a single datum type, and otherwise contains nearly the same\n * structure. The primary sorting key is on the timeseries name, key, and then\n * timestamp, so that all timeseries from the same schema are grouped, followed\n * by all samples from the same timeseries.\n *\n * This reflects that one usually looks up the _key_ in one or more field table,\n * and then uses that to index quickly into the measurements tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Bool)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29712025-04-30T06:26:22.836ZDEBGtest_disk_expunge (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "dc879631-7f6b-4ce9-9913-0934840295cb", "content-length": "133", "date": "Wed, 30 Apr 2025 06:26:22 GMT"} })
29722025-04-30T06:26:22.837ZWARNtest_disk_expunge (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 114.767476ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"dc879631-7f6b-4ce9-9913-0934840295cb\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:26:22 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"dc879631-7f6b-4ce9-9913-0934840295cb\\" }" file = oximeter/producer/src/lib.rs:371
29732025-04-30T06:26:22.839ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29742025-04-30T06:26:22.903ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
29752025-04-30T06:26:22.903ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
29762025-04-30T06:26:22.908ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29772025-04-30T06:26:22.908ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:48951 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
29782025-04-30T06:26:22.910ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:48951 req_id = 687f60f5-2598-41de-8597-64070a4b9a3c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
29792025-04-30T06:26:22.912ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1549 local_addr = [::1]:44559 method = GET remote_addr = [::1]:48951 req_id = 687f60f5-2598-41de-8597-64070a4b9a3c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
29802025-04-30T06:26:22.912ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "687f60f5-2598-41de-8597-64070a4b9a3c", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:26:22 GMT"} })
29812025-04-30T06:26:22.916ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
29822025-04-30T06:26:22.916ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
29832025-04-30T06:26:22.916ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
29842025-04-30T06:26:22.921ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
29852025-04-30T06:26:22.921ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
29862025-04-30T06:26:22.922ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29872025-04-30T06:26:22.922ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29882025-04-30T06:26:22.923ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
29892025-04-30T06:26:22.924ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
29902025-04-30T06:26:22.927ZTRACtest_disk_expunge (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 59 b5 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 Y............_cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:62229 req_id = 0a2ef6b2-cb34-4241-a16b-66fe7e7bca43
29912025-04-30T06:26:22.927ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22965,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62229 req_id = 0a2ef6b2-cb34-4241-a16b-66fe7e7bca43
29922025-04-30T06:26:22.927ZDEBGtest_disk_expunge (store): query key key = _crucible-pantry._tcp
29932025-04-30T06:26:22.927ZDEBGtest_disk_expunge (store): query key key = 116bed1f-7c51-4865-8a2e-dc625a6fb25d.host
29942025-04-30T06:26:22.927ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("116bed1f-7c51-4865-8a2e-dc625a6fb25d.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62229 query = LowerQuery { name: LowerName(Name("_crucible-pantry._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 47737, target: Name("116bed1f-7c51-4865-8a2e-dc625a6fb25d.host.control-plane.oxide.internal") })) }] req_id = 0a2ef6b2-cb34-4241-a16b-66fe7e7bca43
29952025-04-30T06:26:22.927ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
29962025-04-30T06:26:22.933ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
29972025-04-30T06:26:22.934ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
29982025-04-30T06:26:22.934ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
29992025-04-30T06:26:22.934ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
30002025-04-30T06:26:22.937ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30012025-04-30T06:26:22.939ZTRACtest_disk_expunge (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 9b 45 01 00 00 01 00 00 00 00 00 01 24 31 31 36 .E..........$116\n0010: 62 65 64 31 66 2d 37 63 35 31 2d 34 38 36 35 2d bed1f-7c51-4865-\n0020: 38 61 32 65 2d 64 63 36 32 35 61 36 66 62 32 35 8a2e-dc625a6fb25\n0030: 64 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 d.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:54500 req_id = 912dde1a-8ac0-4fa1-a987-253c41c8ca34
30022025-04-30T06:26:22.939ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39749,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("116bed1f-7c51-4865-8a2e-dc625a6fb25d.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("116bed1f-7c51-4865-8a2e-dc625a6fb25d.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 49,\n 49,\n 54,\n 98,\n 101,\n 100,\n 49,\n 102,\n 45,\n 55,\n 99,\n 53,\n 49,\n 45,\n 52,\n 56,\n 54,\n 53,\n 45,\n 56,\n 97,\n 50,\n 101,\n 45,\n 100,\n 99,\n 54,\n 50,\n 53,\n 97,\n 54,\n 102,\n 98,\n 50,\n 53,\n 100,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54500 req_id = 912dde1a-8ac0-4fa1-a987-253c41c8ca34
30032025-04-30T06:26:22.939ZDEBGtest_disk_expunge (store): query key key = 116bed1f-7c51-4865-8a2e-dc625a6fb25d.host
30042025-04-30T06:26:22.939ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54500 query = LowerQuery { name: LowerName(Name("116bed1f-7c51-4865-8a2e-dc625a6fb25d.host.control-plane.oxide.internal.")), original: Query { name: Name("116bed1f-7c51-4865-8a2e-dc625a6fb25d.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("116bed1f-7c51-4865-8a2e-dc625a6fb25d.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 912dde1a-8ac0-4fa1-a987-253c41c8ca34
30052025-04-30T06:26:23.013ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30062025-04-30T06:26:23.015ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:33974 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30072025-04-30T06:26:23.017ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:33974 req_id = 129be337-5327-4673-938d-1ea16fdf8f41 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30082025-04-30T06:26:23.019ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 2142 local_addr = [::1]:47737 method = GET remote_addr = [::1]:33974 req_id = 129be337-5327-4673-938d-1ea16fdf8f41 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30092025-04-30T06:26:23.019ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30102025-04-30T06:26:23.522ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30112025-04-30T06:26:23.533ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:51580 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30122025-04-30T06:26:23.533ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:35660 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30132025-04-30T06:26:23.533ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:44119 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30142025-04-30T06:26:23.533ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:62104 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30152025-04-30T06:26:23.534ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:63623 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30162025-04-30T06:26:23.534ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:58071 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30172025-04-30T06:26:23.534ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:49709 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30182025-04-30T06:26:23.536ZDEBGtest_disk_expunge (producer-registration-task): will register with Nexus at 127.0.0.1:34036
30192025-04-30T06:26:23.643ZDEBGtest_disk_expunge (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:34036/metrics/producers
30202025-04-30T06:26:23.648ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30212025-04-30T06:26:23.653ZINFOtest_disk_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:34036 remote_addr = 127.0.0.1:33309
30222025-04-30T06:26:23.653ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:51580 req_id = 73d6d07a-eedf-46f3-98cb-84cff3549a6b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30232025-04-30T06:26:23.654ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:35660 req_id = d796787d-66a3-457d-9ea0-8d0b69388f30 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30242025-04-30T06:26:23.654ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:44119 req_id = b25182fa-902a-42c7-96fd-f67bf77de545 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30252025-04-30T06:26:23.654ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:62104 req_id = 017009af-16c6-473d-bf5b-c87c2e6083be sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30262025-04-30T06:26:23.654ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:63623 req_id = cb51991d-0380-41dc-b4e2-240ba688880f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30272025-04-30T06:26:23.654ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:58071 req_id = 2f29a0a0-2dfb-405f-a232-185338e94eda sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30282025-04-30T06:26:23.655ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:49709 req_id = d4afbac6-0262-44d8-b64d-314a287910cb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30292025-04-30T06:26:23.656ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 2407 local_addr = [::1]:47737 method = GET remote_addr = [::1]:51580 req_id = 73d6d07a-eedf-46f3-98cb-84cff3549a6b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30302025-04-30T06:26:23.656ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 2188 local_addr = [::1]:47737 method = GET remote_addr = [::1]:35660 req_id = d796787d-66a3-457d-9ea0-8d0b69388f30 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30312025-04-30T06:26:23.656ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 2104 local_addr = [::1]:47737 method = GET remote_addr = [::1]:44119 req_id = b25182fa-902a-42c7-96fd-f67bf77de545 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30322025-04-30T06:26:23.656ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 2026 local_addr = [::1]:47737 method = GET remote_addr = [::1]:62104 req_id = 017009af-16c6-473d-bf5b-c87c2e6083be response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30332025-04-30T06:26:23.656ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 1947 local_addr = [::1]:47737 method = GET remote_addr = [::1]:63623 req_id = cb51991d-0380-41dc-b4e2-240ba688880f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30342025-04-30T06:26:23.657ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 2738 local_addr = [::1]:47737 method = GET remote_addr = [::1]:58071 req_id = 2f29a0a0-2dfb-405f-a232-185338e94eda response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30352025-04-30T06:26:23.657ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 2131 local_addr = [::1]:47737 method = GET remote_addr = [::1]:49709 req_id = d4afbac6-0262-44d8-b64d-314a287910cb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30362025-04-30T06:26:23.657ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30372025-04-30T06:26:23.659ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:33309 req_id = 4122e5f8-7a88-42d5-8878-dd6ae3f717db uri = /metrics/producers
30382025-04-30T06:26:23.673ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:33309 req_id = 4122e5f8-7a88-42d5-8878-dd6ae3f717db resource = Database uri = /metrics/producers
30392025-04-30T06:26:23.673ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:33309 req_id = 4122e5f8-7a88-42d5-8878-dd6ae3f717db roles = RoleSet { roles: {} } uri = /metrics/producers
30402025-04-30T06:26:23.674ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:33309 req_id = 4122e5f8-7a88-42d5-8878-dd6ae3f717db resource = Database result = Ok(()) uri = /metrics/producers
30412025-04-30T06:26:23.677ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30422025-04-30T06:26:23.684ZINFOtest_disk_expunge (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 25375 local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:33309 req_id = 4122e5f8-7a88-42d5-8878-dd6ae3f717db response_code = 503 uri = /metrics/producers
30432025-04-30T06:26:23.685ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(String)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30442025-04-30T06:26:23.685ZDEBGtest_disk_expunge (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "4122e5f8-7a88-42d5-8878-dd6ae3f717db", "content-length": "133", "date": "Wed, 30 Apr 2025 06:26:23 GMT"} })
30452025-04-30T06:26:23.687ZWARNtest_disk_expunge (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 964.580533ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"4122e5f8-7a88-42d5-8878-dd6ae3f717db\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:26:23 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"4122e5f8-7a88-42d5-8878-dd6ae3f717db\\" }" file = oximeter/producer/src/lib.rs:371
30462025-04-30T06:26:23.738ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
30472025-04-30T06:26:23.738ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
30482025-04-30T06:26:23.739ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 2.336561856s iteration = 3 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
30492025-04-30T06:26:23.739ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bytes\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Right now we can't unambiguously record a nullable byte array.\n * Arrays cannot be nested in `Nullable()` types, and encoding the array as\n * a string isn't palatable for a few reasons.\n * See: https://github.com/oxidecomputer/omicron/issues/4551 for more\n * details.\n */\n datum Array(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30502025-04-30T06:26:23.757ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativei64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30512025-04-30T06:26:23.765ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativeu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30522025-04-30T06:26:23.769ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30532025-04-30T06:26:23.775ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30542025-04-30T06:26:23.780ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Array types cannot be Nullable, see\n * https://clickhouse.com/docs/en/sql-reference/data-types/nullable\n * for more details.\n *\n * This means we need to use empty arrays to indicate a missing value. This\n * is unfortunate, and at this point relies on the fact that an\n * `oximeter::Histogram` cannot have zero bins. If that changes, we'll need\n * to figure out another way to represent missing samples here.\n */\n bins Array(Int8),\n counts Array(UInt64),\n min Int8,\n max Int8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30552025-04-30T06:26:23.784ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt8),\n counts Array(UInt64),\n min UInt8,\n max UInt8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30562025-04-30T06:26:23.788ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int16),\n counts Array(UInt64),\n min Int16,\n max Int16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30572025-04-30T06:26:23.791ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt16),\n counts Array(UInt64),\n min UInt16,\n max UInt16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30582025-04-30T06:26:23.795ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int32),\n counts Array(UInt64),\n min Int32,\n max Int32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30592025-04-30T06:26:23.805ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt32),\n counts Array(UInt64),\n min UInt32,\n max UInt32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30602025-04-30T06:26:23.824ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int64),\n counts Array(UInt64),\n min Int64,\n max Int64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30612025-04-30T06:26:23.834ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt64),\n counts Array(UInt64),\n min UInt64,\n max UInt64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30622025-04-30T06:26:23.838ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float32),\n counts Array(UInt64),\n min Float32,\n max Float32,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30632025-04-30T06:26:23.842ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float64),\n counts Array(UInt64),\n min Float64,\n max Float64,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30642025-04-30T06:26:23.853ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\n/* The field tables store named dimensions of each timeseries.\n *\n * As with the measurement tables, there is one field table for each field data\n * type. Fields are deduplicated by using the "replacing merge tree", though\n * this behavior **must not** be relied upon for query correctness.\n *\n * The index for the fields differs from the measurements, however. Rows are\n * sorted by timeseries name, then field name, field value, and finally\n * timeseries key. This reflects the most common pattern for looking them up:\n * by field name and possibly value, within a timeseries. The resulting keys are\n * usually then used to look up measurements.\n *\n * NOTE: We may want to consider a secondary index on these tables, sorting by\n * timeseries name and then key, since it would improve lookups where one\n * already has the key. Realistically though, these tables are quite small and\n * so performance benefits will be low in absolute terms.\n *\n * TTL: We use a materialized column to expire old field table records. This\n * column is generated automatically by the database whenever a new row is\n * inserted. It cannot be inserted directly, nor is it returned in a `SELECT *`\n * query. Since these tables are `ReplacingMergeTree`s, that means the last\n * record will remain during a deduplication, which will have the last\n * timestamp. ClickHouse will then expire old data for us, similar to the\n * measurement tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.fields_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Bool,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30652025-04-30T06:26:23.859ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30662025-04-30T06:26:23.864ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30672025-04-30T06:26:23.868ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30682025-04-30T06:26:23.872ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30692025-04-30T06:26:23.876ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30702025-04-30T06:26:23.880ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30712025-04-30T06:26:23.888ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30722025-04-30T06:26:23.892ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30732025-04-30T06:26:23.897ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_ipaddr\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value IPv6,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30742025-04-30T06:26:23.920ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value String,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30752025-04-30T06:26:23.931ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_uuid\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UUID,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30762025-04-30T06:26:23.937ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = \n\n/* The timeseries schema table stores the extracted schema for the samples\n * oximeter collects.\n */\nCREATE TABLE IF NOT EXISTS oximeter.timeseries_schema\n(\n timeseries_name String,\n fields Nested(\n name String,\n type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'IpAddr' = 3,\n 'String' = 4,\n 'Uuid' = 6,\n 'I8' = 7,\n 'U8' = 8,\n 'I16' = 9,\n 'U16' = 10,\n 'I32' = 11,\n 'U32' = 12,\n 'U64' = 13\n ),\n source Enum(\n 'Target' = 1,\n 'Metric' = 2\n )\n ),\n datum_type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'F64' = 3,\n 'String' = 4,\n 'Bytes' = 5,\n 'CumulativeI64' = 6,\n 'CumulativeF64' = 7,\n 'HistogramI64' = 8,\n 'HistogramF64' = 9,\n 'I8' = 10,\n 'U8' = 11,\n 'I16' = 12,\n 'U16' = 13,\n 'I32' = 14,\n 'U32' = 15,\n 'U64' = 16,\n 'F32' = 17,\n 'CumulativeU64' = 18,\n 'CumulativeF32' = 19,\n 'HistogramI8' = 20,\n 'HistogramU8' = 21,\n 'HistogramI16' = 22,\n 'HistogramU16' = 23,\n 'HistogramI32' = 24,\n 'HistogramU32' = 25,\n 'HistogramU64' = 26,\n 'HistogramF32' = 27\n ),\n created DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, fields.name)
30772025-04-30T06:26:23.940ZINFOtest_disk_expunge (clickhouse-client): inserting current version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:825 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 version = 14
30782025-04-30T06:26:23.940ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = INSERT INTO oximeter.version (*) VALUES (14, now());
30792025-04-30T06:26:23.949ZDEBGtest_disk_expunge (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 7c77095d-cf21-49ff-a074-621aefaa6d37
30802025-04-30T06:26:23.950ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:55725 method = GET path = /info versions = all
30812025-04-30T06:26:23.950ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:55725 method = GET path = /producers versions = all
30822025-04-30T06:26:23.950ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:55725 method = DELETE path = /producers/{producer_id} versions = all
30832025-04-30T06:26:23.950ZDEBGtest_disk_expunge (dropshot): registered endpoint local_addr = [::1]:55725 method = GET path = /producers/{producer_id} versions = all
30842025-04-30T06:26:23.950ZINFOtest_disk_expunge (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [::1]:55725
30852025-04-30T06:26:23.953ZDEBGtest_disk_expunge (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:55725
30862025-04-30T06:26:23.954ZDEBGtest_disk_expunge (oximeter): registered USDT probes
30872025-04-30T06:26:23.955ZDEBGtest_disk_expunge (oximeter): contacting nexus
30882025-04-30T06:26:24.758ZDEBGtest_disk_expunge (oximeter): client request body = Some(Body) method = POST uri = http://127.0.0.1:34036/metrics/collectors
30892025-04-30T06:26:24.759ZDEBGtest_disk_expunge (producer-registration-task): will register with Nexus at 127.0.0.1:34036
30902025-04-30T06:26:24.833ZDEBGtest_disk_expunge (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:34036/metrics/producers
30912025-04-30T06:26:24.834ZINFOtest_disk_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:34036 remote_addr = 127.0.0.1:58228
30922025-04-30T06:26:24.834ZINFOtest_disk_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:34036 remote_addr = 127.0.0.1:60113
30932025-04-30T06:26:24.835ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:60113 req_id = 332d5792-e0b0-4522-99a4-5107b78db119 uri = /metrics/producers
30942025-04-30T06:26:24.836ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:60113 req_id = 332d5792-e0b0-4522-99a4-5107b78db119 resource = Database uri = /metrics/producers
30952025-04-30T06:26:24.836ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:60113 req_id = 332d5792-e0b0-4522-99a4-5107b78db119 roles = RoleSet { roles: {} } uri = /metrics/producers
30962025-04-30T06:26:24.837ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:60113 req_id = 332d5792-e0b0-4522-99a4-5107b78db119 resource = Database result = Ok(()) uri = /metrics/producers
30972025-04-30T06:26:24.838ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:58228 req_id = 49e90be9-b42e-4a44-b2c2-97ca874a16e9 uri = /metrics/collectors
30982025-04-30T06:26:24.839ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:58228 req_id = 49e90be9-b42e-4a44-b2c2-97ca874a16e9 resource = Database uri = /metrics/collectors
30992025-04-30T06:26:24.839ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:58228 req_id = 49e90be9-b42e-4a44-b2c2-97ca874a16e9 roles = RoleSet { roles: {} } uri = /metrics/collectors
31002025-04-30T06:26:24.839ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:58228 req_id = 49e90be9-b42e-4a44-b2c2-97ca874a16e9 resource = Database result = Ok(()) uri = /metrics/collectors
31012025-04-30T06:26:24.845ZINFOtest_disk_expunge (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 9558 local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:60113 req_id = 332d5792-e0b0-4522-99a4-5107b78db119 response_code = 503 uri = /metrics/producers
31022025-04-30T06:26:24.845ZDEBGtest_disk_expunge (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "332d5792-e0b0-4522-99a4-5107b78db119", "content-length": "133", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
31032025-04-30T06:26:24.846ZWARNtest_disk_expunge (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 2.123983107s error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"332d5792-e0b0-4522-99a4-5107b78db119\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:26:24 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"332d5792-e0b0-4522-99a4-5107b78db119\\" }" file = oximeter/producer/src/lib.rs:371
31042025-04-30T06:26:24.849ZINFOtest_disk_expunge (nexus): registered new oximeter metric collection server address = [::1]:55725 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31052025-04-30T06:26:24.850ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 11523 local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:58228 req_id = 49e90be9-b42e-4a44-b2c2-97ca874a16e9 response_code = 204 uri = /metrics/collectors
31062025-04-30T06:26:24.850ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors", status: 204, headers: {"x-request-id": "49e90be9-b42e-4a44-b2c2-97ca874a16e9", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
31072025-04-30T06:26:24.850ZINFOtest_disk_expunge (oximeter): oximeter registered with nexus file = oximeter/collector/src/lib.rs:392 id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31082025-04-30T06:26:24.850ZDEBGtest_disk_expunge (init_with_steps): Running step start_producer_server
31092025-04-30T06:26:24.851ZDEBGtest_disk_expunge: Starting test metric Producer Server
31102025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = Action
31112025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = AnyActor
31122025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = AuthenticatedActor
31132025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = BlueprintConfig
31142025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = Database
31152025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = DnsConfig
31162025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = Fleet
31172025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = Inventory
31182025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = IpPoolList
31192025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = ConsoleSessionList
31202025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = DeviceAuthRequestList
31212025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = SiloCertificateList
31222025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = SiloIdentityProviderList
31232025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = SiloUserList
31242025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = TargetReleaseConfig
31252025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = WebhookEventClassList
31262025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = Project
31272025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = Disk
31282025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = Snapshot
31292025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = ProjectImage
31302025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = AffinityGroup
31312025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = AntiAffinityGroup
31322025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = Instance
31332025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = IpPool
31342025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = InstanceNetworkInterface
31352025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = Vpc
31362025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = VpcRouter
31372025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = InternetGateway
31382025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = InternetGatewayIpPool
31392025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = InternetGatewayIpAddress
31402025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = RouterRoute
31412025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = VpcSubnet
31422025-04-30T06:26:24.853ZINFOtest_disk_expunge: registering Oso class class = FloatingIp
31432025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = Image
31442025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = SiloImage
31452025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = AddressLot
31462025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = Blueprint
31472025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = LoopbackAddress
31482025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = Certificate
31492025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = ConsoleSession
31502025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = DeviceAuthRequest
31512025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = DeviceAccessToken
31522025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = PhysicalDisk
31532025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = Rack
31542025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = RoleBuiltin
31552025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = SshKey
31562025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = Silo
31572025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = SiloUser
31582025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = SiloGroup
31592025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = SupportBundle
31602025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = IdentityProvider
31612025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = SamlIdentityProvider
31622025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = Sled
31632025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = TufRepo
31642025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = TufArtifact
31652025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = WebhookEvent
31662025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = WebhookReceiver
31672025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = WebhookSecret
31682025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = Zpool
31692025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = Service
31702025-04-30T06:26:24.854ZINFOtest_disk_expunge: registering Oso class class = UserBuiltin
31712025-04-30T06:26:24.854ZINFOtest_disk_expunge: full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
31722025-04-30T06:26:24.871ZDEBGtest_disk_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:57310 method = GET path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31732025-04-30T06:26:24.871ZDEBGtest_disk_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:57310 method = POST path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31742025-04-30T06:26:24.871ZDEBGtest_disk_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:57310 method = DELETE path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31752025-04-30T06:26:24.871ZDEBGtest_disk_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:57310 method = GET path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31762025-04-30T06:26:24.871ZDEBGtest_disk_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:57310 method = GET path = /crucible/0/regions/{id}/snapshots server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31772025-04-30T06:26:24.871ZDEBGtest_disk_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:57310 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31782025-04-30T06:26:24.871ZDEBGtest_disk_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:57310 method = GET path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31792025-04-30T06:26:24.871ZDEBGtest_disk_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:57310 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31802025-04-30T06:26:24.871ZDEBGtest_disk_expunge (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:57310 method = POST path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
31812025-04-30T06:26:24.871ZINFOtest_disk_expunge (Simulated CrucibleAgent Dropshot Server): listening kind = storage local_addr = [::1]:57310 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31822025-04-30T06:26:24.872ZDEBGtest_disk_expunge (Simulated CrucibleAgent Dropshot Server): successfully registered DTrace USDT probes kind = storage local_addr = [::1]:57310 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31832025-04-30T06:26:24.872ZINFOtest_disk_expunge (SledAgent): Created Simulated Crucible Server address = [::1]:57310 kind = storage server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
31842025-04-30T06:26:24.872ZINFOtest_disk_expunge (nexus): inserting test physical disk model = test-model physical_disk_id = ad764934-9597-43ac-998a-e59d6e43a547 serial = totally-unique-serial: ad764934-9597-43ac-998a-e59d6e43a547 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
31852025-04-30T06:26:24.872ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
31862025-04-30T06:26:24.872ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
31872025-04-30T06:26:24.873ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
31882025-04-30T06:26:24.874ZINFOtest_disk_expunge (oximeter-agent): starting refresh list task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:559
31892025-04-30T06:26:24.930ZINFOtest_disk_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:34036 remote_addr = 127.0.0.1:56517
31902025-04-30T06:26:24.930ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
31912025-04-30T06:26:24.933ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31922025-04-30T06:26:24.933ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d3c4e9a5-c06d-485b-88f0-dbafea3a5b5f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31932025-04-30T06:26:24.934ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:56517 req_id = 36204eb9-c802-4d82-b429-e27247ab372d uri = /metrics/producers
31942025-04-30T06:26:24.936ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d3c4e9a5-c06d-485b-88f0-dbafea3a5b5f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31952025-04-30T06:26:24.936ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d3c4e9a5-c06d-485b-88f0-dbafea3a5b5f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31962025-04-30T06:26:24.937ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d3c4e9a5-c06d-485b-88f0-dbafea3a5b5f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31972025-04-30T06:26:24.937ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:56517 req_id = 36204eb9-c802-4d82-b429-e27247ab372d resource = Database uri = /metrics/producers
31982025-04-30T06:26:24.937ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:56517 req_id = 36204eb9-c802-4d82-b429-e27247ab372d roles = RoleSet { roles: {} } uri = /metrics/producers
31992025-04-30T06:26:24.938ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:56517 req_id = 36204eb9-c802-4d82-b429-e27247ab372d resource = Database result = Ok(()) uri = /metrics/producers
32002025-04-30T06:26:24.939ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32012025-04-30T06:26:24.939ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32022025-04-30T06:26:24.940ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32032025-04-30T06:26:24.946ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 12309 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d3c4e9a5-c06d-485b-88f0-dbafea3a5b5f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
32042025-04-30T06:26:24.947ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3c4e9a5-c06d-485b-88f0-dbafea3a5b5f", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
32052025-04-30T06:26:24.947ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 0 n_pruned_tasks = 0
32062025-04-30T06:26:24.947ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
32072025-04-30T06:26:24.947ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32082025-04-30T06:26:24.947ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32092025-04-30T06:26:24.947ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32102025-04-30T06:26:24.948ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32112025-04-30T06:26:24.957ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32122025-04-30T06:26:24.962ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
32132025-04-30T06:26:24.963ZTRACtest_disk_expunge (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
32142025-04-30T06:26:24.963ZTRACtest_disk_expunge (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
32152025-04-30T06:26:25.003ZINFOtest_disk_expunge (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
32162025-04-30T06:26:25.004ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 69497 local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:56517 req_id = 36204eb9-c802-4d82-b429-e27247ab372d response_code = 201 uri = /metrics/producers
32172025-04-30T06:26:25.097ZINFOtest_disk_expunge (nexus): upserting zpool physical_disk_id = ad764934-9597-43ac-998a-e59d6e43a547 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 47a6475f-ad96-4f0e-a278-6f98b7783e55
32182025-04-30T06:26:25.097ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32192025-04-30T06:26:25.097ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32202025-04-30T06:26:25.099ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32212025-04-30T06:26:25.100ZTRACtest_disk_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
32222025-04-30T06:26:25.100ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32232025-04-30T06:26:25.100ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32242025-04-30T06:26:25.100ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32252025-04-30T06:26:25.101ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32262025-04-30T06:26:25.120ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32272025-04-30T06:26:25.123ZDEBGtest_disk_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
32282025-04-30T06:26:25.142ZINFOtest_disk_expunge (nexus): upserting Crucible dataset dataset_id = 9c7b063f-dcc4-4eba-916e-a124eb604d4e zpool_id = 47a6475f-ad96-4f0e-a278-6f98b7783e55
32292025-04-30T06:26:25.167ZINFOtest_disk_expunge (nexus): inserting test physical disk model = test-model physical_disk_id = ad764934-9597-43ac-998a-e59d6e43a547 serial = totally-unique-serial: ad764934-9597-43ac-998a-e59d6e43a547 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
32302025-04-30T06:26:25.167ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32312025-04-30T06:26:25.167ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32322025-04-30T06:26:25.169ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32332025-04-30T06:26:25.185ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: ad764934-9597-43ac-998a-e59d6e43a547 (physical_disk), lookup_type: ById(ad764934-9597-43ac-998a-e59d6e43a547) }
32342025-04-30T06:26:25.186ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32352025-04-30T06:26:25.186ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32362025-04-30T06:26:25.186ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32372025-04-30T06:26:25.187ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32382025-04-30T06:26:25.201ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32392025-04-30T06:26:25.208ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: ad764934-9597-43ac-998a-e59d6e43a547 (physical_disk), lookup_type: ById(ad764934-9597-43ac-998a-e59d6e43a547) } result = Ok(())
32402025-04-30T06:26:25.209ZINFOtest_disk_expunge (nexus): upserting zpool physical_disk_id = ad764934-9597-43ac-998a-e59d6e43a547 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 47a6475f-ad96-4f0e-a278-6f98b7783e55
32412025-04-30T06:26:25.209ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32422025-04-30T06:26:25.209ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32432025-04-30T06:26:25.209ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32442025-04-30T06:26:25.217ZTRACtest_disk_expunge (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
32452025-04-30T06:26:25.217ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32462025-04-30T06:26:25.217ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32472025-04-30T06:26:25.217ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32482025-04-30T06:26:25.217ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32492025-04-30T06:26:25.232ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32502025-04-30T06:26:25.235ZDEBGtest_disk_expunge (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
32512025-04-30T06:26:25.241ZTRACtest_disk_expunge (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
32522025-04-30T06:26:25.241ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32532025-04-30T06:26:25.241ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32542025-04-30T06:26:25.241ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32552025-04-30T06:26:25.242ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32562025-04-30T06:26:25.264ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32572025-04-30T06:26:25.280ZDEBGtest_disk_expunge (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
32582025-04-30T06:26:25.280ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32592025-04-30T06:26:25.280ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32602025-04-30T06:26:25.281ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32612025-04-30T06:26:25.320ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
32622025-04-30T06:26:25.320ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32632025-04-30T06:26:25.320ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32642025-04-30T06:26:25.320ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32652025-04-30T06:26:25.321ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32662025-04-30T06:26:25.321ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 4 reason = Signaled
32672025-04-30T06:26:25.321ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
32682025-04-30T06:26:25.321ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32692025-04-30T06:26:25.321ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32702025-04-30T06:26:25.321ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32712025-04-30T06:26:25.321ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32722025-04-30T06:26:25.325ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32732025-04-30T06:26:25.334ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
32742025-04-30T06:26:25.334ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32752025-04-30T06:26:25.334ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32762025-04-30T06:26:25.335ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32772025-04-30T06:26:25.337ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
32782025-04-30T06:26:25.340ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
32792025-04-30T06:26:25.340ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
32802025-04-30T06:26:25.340ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
32812025-04-30T06:26:25.341ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
32822025-04-30T06:26:25.341ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
32832025-04-30T06:26:25.341ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
32842025-04-30T06:26:25.342ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
32852025-04-30T06:26:25.354ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
32862025-04-30T06:26:25.354ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
32872025-04-30T06:26:25.356ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 3e 30 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 >0..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53600 req_id = 53e6f339-a670-41ff-82a2-905dcbe11a87
32882025-04-30T06:26:25.356ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15920,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53600 req_id = 53e6f339-a670-41ff-82a2-905dcbe11a87
32892025-04-30T06:26:25.359ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
32902025-04-30T06:26:25.359ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32912025-04-30T06:26:25.359ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53600 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 53e6f339-a670-41ff-82a2-905dcbe11a87
32922025-04-30T06:26:25.359ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3250, tv_nsec: 158810455 } })
32932025-04-30T06:26:25.359ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
32942025-04-30T06:26:25.360ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c0 b8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50316 req_id = 89210820-5941-482e-b1f1-c17132bd35a6
32952025-04-30T06:26:25.360ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49336,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50316 req_id = 89210820-5941-482e-b1f1-c17132bd35a6
32962025-04-30T06:26:25.360ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
32972025-04-30T06:26:25.360ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50316 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 89210820-5941-482e-b1f1-c17132bd35a6
32982025-04-30T06:26:25.458ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
32992025-04-30T06:26:25.459ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 02 f1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63299 req_id = 6ac23bb1-dc99-47ec-8007-74bedb9faad6
33002025-04-30T06:26:25.459ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 753,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63299 req_id = 6ac23bb1-dc99-47ec-8007-74bedb9faad6
33012025-04-30T06:26:25.460ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
33022025-04-30T06:26:25.460ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 753, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63299 req_id = 6ac23bb1-dc99-47ec-8007-74bedb9faad6
33032025-04-30T06:26:25.460ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
33042025-04-30T06:26:25.461ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
33052025-04-30T06:26:25.461ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
33062025-04-30T06:26:25.467ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "32831bd9-8ade-4950-ab00-63a1bec201ae", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33072025-04-30T06:26:25.468ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
33082025-04-30T06:26:25.490ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eedd6fd5-e306-4e28-8995-6bbe9db14e5f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33092025-04-30T06:26:25.491ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
33102025-04-30T06:26:25.505ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e639889-6b2c-40b1-90d4-b35b848efe78", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33112025-04-30T06:26:25.508ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
33122025-04-30T06:26:25.518ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60bd176a-b738-4bb5-8f8e-cb7ae99a905e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33132025-04-30T06:26:25.518ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
33142025-04-30T06:26:25.522ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d145436d-4a00-40fd-ab5b-3a5d82350148", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33152025-04-30T06:26:25.522ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
33162025-04-30T06:26:25.525ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b588d84d-4c25-41a0-b535-8a5ae8208f04", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33172025-04-30T06:26:25.525ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
33182025-04-30T06:26:25.540ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "871be296-5cbb-41f3-9e89-bb6e90ef819a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33192025-04-30T06:26:25.541ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
33202025-04-30T06:26:25.543ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0721592-d891-4419-b75a-1dfc986bf59a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33212025-04-30T06:26:25.543ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
33222025-04-30T06:26:25.545ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "710a39ae-3c49-413e-94b7-ca0965b373f5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33232025-04-30T06:26:25.545ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
33242025-04-30T06:26:25.549ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7816ff7e-47ce-4d50-b740-fee7eb0835dc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33252025-04-30T06:26:25.549ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
33262025-04-30T06:26:25.550ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3b8fe79-04d0-4ec5-9063-4f04db63fb5a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33272025-04-30T06:26:25.550ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
33282025-04-30T06:26:25.551ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
33292025-04-30T06:26:25.553ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b30cccea-8687-4af9-9835-733c4ab6edd0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33302025-04-30T06:26:25.553ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
33312025-04-30T06:26:25.557ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6743971-13a5-43ca-a1cf-a32123310d50", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33322025-04-30T06:26:25.557ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
33332025-04-30T06:26:25.560ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bd07331-387c-4cc6-a9cf-eeacecad6aa2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33342025-04-30T06:26:25.561ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
33352025-04-30T06:26:25.568ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73c605ea-d097-4c4e-bc37-5bceb7ed4f3c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33362025-04-30T06:26:25.568ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
33372025-04-30T06:26:25.569ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b1f83aa-3c62-4ff2-b86a-20e5dcb6efb8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33382025-04-30T06:26:25.569ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
33392025-04-30T06:26:25.574ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "922d1af2-2fdb-410e-bf5c-614473e486a8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33402025-04-30T06:26:25.575ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
33412025-04-30T06:26:25.577ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c7fd0b9-0c3b-488e-826e-1b0e569874db", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33422025-04-30T06:26:25.577ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
33432025-04-30T06:26:25.579ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b9e0730-0cb4-4904-a07e-57f9d526bc5d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33442025-04-30T06:26:25.579ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
33452025-04-30T06:26:25.580ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19236486-d4c0-47f7-a39a-220d2320d499", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33462025-04-30T06:26:25.581ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
33472025-04-30T06:26:25.582ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fed297b-6d62-4d50-8de7-d32fc724f2c6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33482025-04-30T06:26:25.582ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
33492025-04-30T06:26:25.584ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ce0acae-ca6b-4174-9223-df0ed946ac76", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33502025-04-30T06:26:25.584ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
33512025-04-30T06:26:25.586ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "264d3138-cd36-4ea0-88ab-d78df7ee3572", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33522025-04-30T06:26:25.586ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
33532025-04-30T06:26:25.588ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9930aa39-8d11-4ca5-9047-4b617ea3d6eb", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33542025-04-30T06:26:25.603ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
33552025-04-30T06:26:25.603ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33562025-04-30T06:26:25.604ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33572025-04-30T06:26:25.604ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33582025-04-30T06:26:25.604ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
33592025-04-30T06:26:25.606ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33602025-04-30T06:26:25.611ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33612025-04-30T06:26:25.618ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
33622025-04-30T06:26:25.619ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33632025-04-30T06:26:25.619ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
33642025-04-30T06:26:25.619ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33652025-04-30T06:26:25.620ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2c72eea-25b3-42a0-a0f0-539e83daf30d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33662025-04-30T06:26:25.620ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
33672025-04-30T06:26:25.623ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33682025-04-30T06:26:25.624ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
33692025-04-30T06:26:25.624ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33702025-04-30T06:26:25.626ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ed3770e-be30-4cd3-8f27-c2eb9917ec0d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33712025-04-30T06:26:25.626ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
33722025-04-30T06:26:25.629ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c01c732-efba-42cd-8937-ae2142a4cb7b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33732025-04-30T06:26:25.629ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
33742025-04-30T06:26:25.631ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ede5e909-03f9-4252-a847-a30fd182e2da", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33752025-04-30T06:26:25.631ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
33762025-04-30T06:26:25.633ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "785dc9cd-a57c-4d4a-b25b-bccc686048c4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33772025-04-30T06:26:25.633ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
33782025-04-30T06:26:25.636ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e458db9f-3b62-4982-9963-1c805d9ec504", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33792025-04-30T06:26:25.636ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
33802025-04-30T06:26:25.637ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98416718-6de4-4ec0-9c89-39fce4ed5c5c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33812025-04-30T06:26:25.637ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
33822025-04-30T06:26:25.638ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22b7cc4c-9460-4f4d-b1cf-753690bbed03", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33832025-04-30T06:26:25.638ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
33842025-04-30T06:26:25.640ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ccae898-2731-4c84-9a16-6e5d0cea30c4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33852025-04-30T06:26:25.665ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
33862025-04-30T06:26:25.665ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6504d631-0554-47e5-a71b-2c06648d8fa0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33872025-04-30T06:26:25.665ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
33882025-04-30T06:26:25.665ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1468cbb2-2594-4d51-894c-e76817d4efed", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33892025-04-30T06:26:25.665ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
33902025-04-30T06:26:25.665ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80e5d31e-afe9-4670-b617-05f33f9c88db", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33912025-04-30T06:26:25.665ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
33922025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a8923fe-77b0-4c03-b696-6842e717577a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33932025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
33942025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfe63e3e-b18d-4f66-95fd-8fd10928b4aa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33952025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
33962025-04-30T06:26:25.666ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
33972025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3a3b191-0fe5-4129-a0ae-da1bf6e498b4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
33982025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
33992025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc9ea7e5-e54e-4639-9154-10ea44efb8af", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
34002025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
34012025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e312a3c0-9c4b-4cbe-9eea-9e061806c3d4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
34022025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
34032025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f768cd32-d868-447e-8d8c-071938e1f476", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
34042025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
34052025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "822e3a34-4414-4a41-a819-d4d195bc0ed0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
34062025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
34072025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80f5beb5-824d-4a17-8d53-225593137008", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
34082025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
34092025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f5a5148-b15e-4478-809b-bce76c7fd7f9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
34102025-04-30T06:26:25.666ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
34112025-04-30T06:26:25.666ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34122025-04-30T06:26:25.666ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34132025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34142025-04-30T06:26:25.666ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34152025-04-30T06:26:25.670ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34162025-04-30T06:26:25.675ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
34172025-04-30T06:26:25.675ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
34182025-04-30T06:26:25.675ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34192025-04-30T06:26:25.675ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34202025-04-30T06:26:25.675ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34212025-04-30T06:26:25.676ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34222025-04-30T06:26:25.679ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34232025-04-30T06:26:25.709ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
34242025-04-30T06:26:25.709ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34252025-04-30T06:26:25.709ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34262025-04-30T06:26:25.710ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34272025-04-30T06:26:25.712ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
34282025-04-30T06:26:25.712ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34292025-04-30T06:26:25.712ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34302025-04-30T06:26:25.712ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
34312025-04-30T06:26:25.713ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34322025-04-30T06:26:25.786ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
34332025-04-30T06:26:25.786ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
34342025-04-30T06:26:25.787ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34352025-04-30T06:26:25.792ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34362025-04-30T06:26:25.793ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34372025-04-30T06:26:25.793ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
34382025-04-30T06:26:25.793ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34392025-04-30T06:26:25.793ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:38648 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
34402025-04-30T06:26:25.794ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:38648 req_id = c3b78ef9-64d7-44e5-9a52-1684f14e575e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
34412025-04-30T06:26:25.795ZINFOtest_disk_expunge (dropshot): request completed latency_us = 713 local_addr = [::1]:44559 method = GET remote_addr = [::1]:38648 req_id = c3b78ef9-64d7-44e5-9a52-1684f14e575e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
34422025-04-30T06:26:25.795ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3b78ef9-64d7-44e5-9a52-1684f14e575e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:24 GMT"} })
34432025-04-30T06:26:25.797ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
34442025-04-30T06:26:25.797ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
34452025-04-30T06:26:25.797ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
34462025-04-30T06:26:25.798ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34472025-04-30T06:26:25.799ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34482025-04-30T06:26:25.799ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34492025-04-30T06:26:25.799ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34502025-04-30T06:26:25.799ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34512025-04-30T06:26:25.799ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34522025-04-30T06:26:25.799ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
34532025-04-30T06:26:25.800ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34542025-04-30T06:26:25.804ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34552025-04-30T06:26:25.808ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34562025-04-30T06:26:25.809ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34572025-04-30T06:26:25.809ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34582025-04-30T06:26:25.809ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34592025-04-30T06:26:25.862ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
34602025-04-30T06:26:25.863ZDEBGtest_disk_expunge (producer-registration-task): will register with Nexus at 127.0.0.1:34036
34612025-04-30T06:26:25.938ZDEBGtest_disk_expunge (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:34036/metrics/producers
34622025-04-30T06:26:25.939ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
34632025-04-30T06:26:25.939ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34642025-04-30T06:26:25.939ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34652025-04-30T06:26:25.939ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
34662025-04-30T06:26:25.940ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34672025-04-30T06:26:25.940ZINFOtest_disk_expunge (dropshot_internal): accepted connection local_addr = 127.0.0.1:34036 remote_addr = 127.0.0.1:46563
34682025-04-30T06:26:25.943ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:46563 req_id = daf667a0-bddf-4516-9392-d2cce5cb351e uri = /metrics/producers
34692025-04-30T06:26:25.943ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:46563 req_id = daf667a0-bddf-4516-9392-d2cce5cb351e resource = Database uri = /metrics/producers
34702025-04-30T06:26:25.943ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:46563 req_id = daf667a0-bddf-4516-9392-d2cce5cb351e roles = RoleSet { roles: {} } uri = /metrics/producers
34712025-04-30T06:26:25.944ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:46563 req_id = daf667a0-bddf-4516-9392-d2cce5cb351e resource = Database result = Ok(()) uri = /metrics/producers
34722025-04-30T06:26:25.946ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34732025-04-30T06:26:25.953ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34742025-04-30T06:26:25.953ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34752025-04-30T06:26:25.953ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
34762025-04-30T06:26:25.954ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34772025-04-30T06:26:25.957ZTRACtest_disk_expunge (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
34782025-04-30T06:26:25.958ZTRACtest_disk_expunge (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
34792025-04-30T06:26:25.993ZINFOtest_disk_expunge (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
34802025-04-30T06:26:25.993ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 50608 local_addr = 127.0.0.1:34036 method = POST remote_addr = 127.0.0.1:46563 req_id = daf667a0-bddf-4516-9392-d2cce5cb351e response_code = 201 uri = /metrics/producers
34812025-04-30T06:26:25.994ZDEBGtest_disk_expunge (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "daf667a0-bddf-4516-9392-d2cce5cb351e", "content-length": "41", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
34822025-04-30T06:26:25.996ZDEBGtest_disk_expunge (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
34832025-04-30T06:26:26.005ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34842025-04-30T06:26:26.005ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
34852025-04-30T06:26:26.006ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34862025-04-30T06:26:26.006ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
34872025-04-30T06:26:26.007ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
34882025-04-30T06:26:26.007ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 685.816926ms iteration = 4 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
34892025-04-30T06:26:26.007ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 5 reason = Signaled
34902025-04-30T06:26:26.007ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34912025-04-30T06:26:26.007ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34922025-04-30T06:26:26.007ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34932025-04-30T06:26:26.007ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34942025-04-30T06:26:26.007ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34952025-04-30T06:26:26.010ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34962025-04-30T06:26:26.014ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34972025-04-30T06:26:26.015ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34982025-04-30T06:26:26.015ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34992025-04-30T06:26:26.015ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35002025-04-30T06:26:26.020ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
35012025-04-30T06:26:26.020ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
35022025-04-30T06:26:26.029ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 1e 4f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .O..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50007 req_id = 2d592c73-8a4c-4c16-b7f1-526d0a4bdd4a
35032025-04-30T06:26:26.029ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7759,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50007 req_id = 2d592c73-8a4c-4c16-b7f1-526d0a4bdd4a
35042025-04-30T06:26:26.029ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
35052025-04-30T06:26:26.029ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
35062025-04-30T06:26:26.030ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50007 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2d592c73-8a4c-4c16-b7f1-526d0a4bdd4a
35072025-04-30T06:26:26.031ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3250, tv_nsec: 830125666 } })
35082025-04-30T06:26:26.031ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
35092025-04-30T06:26:26.046ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bd 15 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62581 req_id = 1454f48c-b5e3-4ca4-8d5a-dae1cfda119b
35102025-04-30T06:26:26.046ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48405,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62581 req_id = 1454f48c-b5e3-4ca4-8d5a-dae1cfda119b
35112025-04-30T06:26:26.046ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
35122025-04-30T06:26:26.047ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62581 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1454f48c-b5e3-4ca4-8d5a-dae1cfda119b
35132025-04-30T06:26:26.100ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
35142025-04-30T06:26:26.104ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: cc a7 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50814 req_id = 967b42e2-b886-4214-9774-7251ec031bed
35152025-04-30T06:26:26.104ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52391,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50814 req_id = 967b42e2-b886-4214-9774-7251ec031bed
35162025-04-30T06:26:26.104ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
35172025-04-30T06:26:26.105ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52391, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50814 req_id = 967b42e2-b886-4214-9774-7251ec031bed
35182025-04-30T06:26:26.105ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
35192025-04-30T06:26:26.105ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
35202025-04-30T06:26:26.105ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
35212025-04-30T06:26:26.124ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4c0c150-d511-4023-9944-68d5fd39d527", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35222025-04-30T06:26:26.125ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
35232025-04-30T06:26:26.131ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9af1579c-debc-405f-bf30-b104acde0c27", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35242025-04-30T06:26:26.131ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
35252025-04-30T06:26:26.134ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b308955b-5cb6-4d8f-a318-e2fdde200abb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35262025-04-30T06:26:26.134ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
35272025-04-30T06:26:26.143ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
35282025-04-30T06:26:26.150ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "222bb132-5584-40d4-b100-f028147bccc6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35292025-04-30T06:26:26.150ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
35302025-04-30T06:26:26.152ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4ecc6fb-6a78-43c5-a54c-0b09d350bd2d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35312025-04-30T06:26:26.152ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
35322025-04-30T06:26:26.154ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "544f5772-4488-4f69-ad97-f9a37ca91ae4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35332025-04-30T06:26:26.155ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
35342025-04-30T06:26:26.159ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31482871-5691-4005-8153-7a96e0bcdb67", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35352025-04-30T06:26:26.159ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
35362025-04-30T06:26:26.162ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2154e26e-22d3-4845-863c-d08e559cade4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35372025-04-30T06:26:26.162ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
35382025-04-30T06:26:26.164ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85e6f91e-1c93-431a-bf60-2e73c5f177eb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35392025-04-30T06:26:26.164ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
35402025-04-30T06:26:26.167ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d632b3b0-2de5-4cb9-8cec-45d8c3522f20", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35412025-04-30T06:26:26.167ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
35422025-04-30T06:26:26.169ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc4621e5-bad4-46ac-8cae-8161e959e5f7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35432025-04-30T06:26:26.169ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
35442025-04-30T06:26:26.170ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da7e0864-df72-4ebb-88a4-7aded644f36d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35452025-04-30T06:26:26.170ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
35462025-04-30T06:26:26.172ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0d9c5d4-1d5b-46c8-b8ab-53f985d35398", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35472025-04-30T06:26:26.172ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
35482025-04-30T06:26:26.174ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80a33f49-94e6-4c19-9f61-9743b050697a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35492025-04-30T06:26:26.174ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
35502025-04-30T06:26:26.177ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed80ba57-d8b6-4a19-aece-b4fec3837b8a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35512025-04-30T06:26:26.177ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
35522025-04-30T06:26:26.180ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c351214-fdf3-438c-b579-8246ad7ffb78", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35532025-04-30T06:26:26.180ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
35542025-04-30T06:26:26.184ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a28444d7-f490-4b3c-8090-8c6b9a470335", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35552025-04-30T06:26:26.184ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
35562025-04-30T06:26:26.185ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1b65b1e-dae3-4d50-bf0d-32f53d40b148", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35572025-04-30T06:26:26.185ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
35582025-04-30T06:26:26.188ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb520441-3dfd-4a26-8d2c-e76682686810", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35592025-04-30T06:26:26.188ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
35602025-04-30T06:26:26.189ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17fdebff-d47d-442f-bdfd-542857215074", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35612025-04-30T06:26:26.189ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
35622025-04-30T06:26:26.190ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95fa7c18-759d-4c8b-aa86-69f0f9600e7a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35632025-04-30T06:26:26.190ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
35642025-04-30T06:26:26.191ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "614b60bb-d167-4ed8-8e37-2af5e23b0c51", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35652025-04-30T06:26:26.191ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
35662025-04-30T06:26:26.192ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ef8ea26-a465-43be-8353-07803bf8be15", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35672025-04-30T06:26:26.192ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
35682025-04-30T06:26:26.194ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3f47a12-dcb8-4bee-89f0-da3d2eb3d0c1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35692025-04-30T06:26:26.194ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
35702025-04-30T06:26:26.195ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06f7b709-4b6e-4847-8b13-b28138db3bf0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35712025-04-30T06:26:26.196ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
35722025-04-30T06:26:26.197ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "666cdae4-fb15-4557-9626-de949d826659", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35732025-04-30T06:26:26.197ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
35742025-04-30T06:26:26.198ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35752025-04-30T06:26:26.199ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35762025-04-30T06:26:26.199ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35772025-04-30T06:26:26.199ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
35782025-04-30T06:26:26.199ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35792025-04-30T06:26:26.202ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f894489-deca-4824-bd42-ac79f411d0e2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35802025-04-30T06:26:26.202ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
35812025-04-30T06:26:26.202ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35822025-04-30T06:26:26.208ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35832025-04-30T06:26:26.208ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35842025-04-30T06:26:26.208ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
35852025-04-30T06:26:26.208ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35862025-04-30T06:26:26.210ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34d2a649-51a0-43e8-81ab-2eae375b339e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35872025-04-30T06:26:26.211ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
35882025-04-30T06:26:26.211ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35892025-04-30T06:26:26.211ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
35902025-04-30T06:26:26.211ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35912025-04-30T06:26:26.213ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1d412f2-a7ae-46ed-9e7c-87fba6f5ab35", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35922025-04-30T06:26:26.213ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
35932025-04-30T06:26:26.215ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c129784b-d9e2-4409-82ee-ec2cfddfdd31", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35942025-04-30T06:26:26.216ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
35952025-04-30T06:26:26.217ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45ededc7-3337-4eb0-872b-07d7f6556b6b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35962025-04-30T06:26:26.217ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
35972025-04-30T06:26:26.218ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a8daedc-ef0f-4f5e-be1c-191fcc1213f4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
35982025-04-30T06:26:26.218ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
35992025-04-30T06:26:26.220ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c4ef473-8b67-4482-9c1c-fbd8c18aea46", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36002025-04-30T06:26:26.220ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
36012025-04-30T06:26:26.220ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b55bc7d3-0446-4ad4-b54c-604069620225", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36022025-04-30T06:26:26.220ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
36032025-04-30T06:26:26.221ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de12abe6-08ac-4137-ace3-b7acba778a82", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36042025-04-30T06:26:26.222ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
36052025-04-30T06:26:26.224ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f26a4c7d-26ed-4ef3-87f2-4fb10611f682", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36062025-04-30T06:26:26.224ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
36072025-04-30T06:26:26.226ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23c488ab-abc6-4f99-8ab1-037cf9c8753d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36082025-04-30T06:26:26.226ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
36092025-04-30T06:26:26.229ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98a7ba92-5c73-424b-ad90-07a821b8d789", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36102025-04-30T06:26:26.229ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
36112025-04-30T06:26:26.233ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "021cb7a1-11f1-4b4a-8bf9-23d7ba3b85a8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36122025-04-30T06:26:26.233ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
36132025-04-30T06:26:26.237ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
36142025-04-30T06:26:26.240ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc5ac034-ab70-490d-8e37-15e880d9aea4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36152025-04-30T06:26:26.241ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
36162025-04-30T06:26:26.244ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "429cb1a0-5d1d-4576-bd18-89c1f2624227", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36172025-04-30T06:26:26.244ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
36182025-04-30T06:26:26.245ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "406a36e9-3551-445f-b06f-9c207419a14e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36192025-04-30T06:26:26.245ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
36202025-04-30T06:26:26.247ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6154201-bc7a-48c3-92cf-9786338db8e7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36212025-04-30T06:26:26.247ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
36222025-04-30T06:26:26.248ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b4ad7fc-b033-4d00-b253-e71dc643bd08", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36232025-04-30T06:26:26.248ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
36242025-04-30T06:26:26.249ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c250a4f2-6600-49fb-aa90-f05f477bfde7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36252025-04-30T06:26:26.249ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
36262025-04-30T06:26:26.249ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36272025-04-30T06:26:26.250ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36282025-04-30T06:26:26.250ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36292025-04-30T06:26:26.251ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36302025-04-30T06:26:26.254ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36312025-04-30T06:26:26.261ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
36322025-04-30T06:26:26.261ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
36332025-04-30T06:26:26.261ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36342025-04-30T06:26:26.261ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36352025-04-30T06:26:26.261ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36362025-04-30T06:26:26.262ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36372025-04-30T06:26:26.265ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36382025-04-30T06:26:26.271ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
36392025-04-30T06:26:26.272ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36402025-04-30T06:26:26.272ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36412025-04-30T06:26:26.272ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36422025-04-30T06:26:26.345ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
36432025-04-30T06:26:26.345ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
36442025-04-30T06:26:26.346ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36452025-04-30T06:26:26.346ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36462025-04-30T06:26:26.346ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36472025-04-30T06:26:26.346ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
36482025-04-30T06:26:26.347ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36492025-04-30T06:26:26.347ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:36355 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
36502025-04-30T06:26:26.348ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:36355 req_id = 8fdf3beb-b6d2-4ce3-afcf-2c337e40be6b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
36512025-04-30T06:26:26.349ZINFOtest_disk_expunge (dropshot): request completed latency_us = 617 local_addr = [::1]:44559 method = GET remote_addr = [::1]:36355 req_id = 8fdf3beb-b6d2-4ce3-afcf-2c337e40be6b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
36522025-04-30T06:26:26.349ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fdf3beb-b6d2-4ce3-afcf-2c337e40be6b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
36532025-04-30T06:26:26.351ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
36542025-04-30T06:26:26.351ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
36552025-04-30T06:26:26.351ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
36562025-04-30T06:26:26.353ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36572025-04-30T06:26:26.353ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36582025-04-30T06:26:26.353ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36592025-04-30T06:26:26.353ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36602025-04-30T06:26:26.354ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36612025-04-30T06:26:26.354ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36622025-04-30T06:26:26.363ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36632025-04-30T06:26:26.363ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36642025-04-30T06:26:26.363ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
36652025-04-30T06:26:26.363ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36662025-04-30T06:26:26.365ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36672025-04-30T06:26:26.370ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36682025-04-30T06:26:26.370ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36692025-04-30T06:26:26.370ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36702025-04-30T06:26:26.371ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36712025-04-30T06:26:26.373ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36722025-04-30T06:26:26.373ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
36732025-04-30T06:26:26.374ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36742025-04-30T06:26:26.459ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
36752025-04-30T06:26:26.459ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
36762025-04-30T06:26:26.460ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
36772025-04-30T06:26:26.460ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 452.998149ms iteration = 5 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
36782025-04-30T06:26:26.460ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 6 reason = Signaled
36792025-04-30T06:26:26.460ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36802025-04-30T06:26:26.460ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36812025-04-30T06:26:26.460ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36822025-04-30T06:26:26.460ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36832025-04-30T06:26:26.461ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36842025-04-30T06:26:26.466ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36852025-04-30T06:26:26.473ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36862025-04-30T06:26:26.473ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36872025-04-30T06:26:26.473ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36882025-04-30T06:26:26.473ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36892025-04-30T06:26:26.476ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
36902025-04-30T06:26:26.476ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
36912025-04-30T06:26:26.477ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 16 be 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61973 req_id = 2d9beda2-5202-497f-a8bf-90283cb7b644
36922025-04-30T06:26:26.477ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5822,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61973 req_id = 2d9beda2-5202-497f-a8bf-90283cb7b644
36932025-04-30T06:26:26.478ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
36942025-04-30T06:26:26.478ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36952025-04-30T06:26:26.478ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61973 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2d9beda2-5202-497f-a8bf-90283cb7b644
36962025-04-30T06:26:26.478ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3251, tv_nsec: 277687915 } })
36972025-04-30T06:26:26.478ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
36982025-04-30T06:26:26.479ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 26 84 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e &...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58955 req_id = da7f49a5-2395-4679-b65e-8887bde7ddea
36992025-04-30T06:26:26.479ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9860,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58955 req_id = da7f49a5-2395-4679-b65e-8887bde7ddea
37002025-04-30T06:26:26.479ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
37012025-04-30T06:26:26.479ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58955 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = da7f49a5-2395-4679-b65e-8887bde7ddea
37022025-04-30T06:26:26.549ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
37032025-04-30T06:26:26.549ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37042025-04-30T06:26:26.549ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37052025-04-30T06:26:26.549ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37062025-04-30T06:26:26.549ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
37072025-04-30T06:26:26.550ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37082025-04-30T06:26:26.551ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 19 ea 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64874 req_id = 078c371e-cbb4-489d-9483-200788e46e0d
37092025-04-30T06:26:26.551ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6634,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64874 req_id = 078c371e-cbb4-489d-9483-200788e46e0d
37102025-04-30T06:26:26.552ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
37112025-04-30T06:26:26.552ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6634, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64874 req_id = 078c371e-cbb4-489d-9483-200788e46e0d
37122025-04-30T06:26:26.553ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
37132025-04-30T06:26:26.553ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
37142025-04-30T06:26:26.553ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
37152025-04-30T06:26:26.555ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37162025-04-30T06:26:26.570ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37172025-04-30T06:26:26.570ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37182025-04-30T06:26:26.570ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
37192025-04-30T06:26:26.571ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37202025-04-30T06:26:26.578ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37212025-04-30T06:26:26.578ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
37222025-04-30T06:26:26.580ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37232025-04-30T06:26:26.580ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ad8d228-495d-4ab9-bf38-62400e179ca1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37242025-04-30T06:26:26.580ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
37252025-04-30T06:26:26.584ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1740c1bf-1fa6-40e5-b88d-646de7d975fe", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37262025-04-30T06:26:26.586ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
37272025-04-30T06:26:26.599ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "397f8001-4606-4553-82e4-539be97b2df0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37282025-04-30T06:26:26.599ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
37292025-04-30T06:26:26.602ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9270f66b-3550-4ca1-9db9-1a474950523f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37302025-04-30T06:26:26.602ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
37312025-04-30T06:26:26.608ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce9abe25-56bf-469f-870e-1a446fc53102", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37322025-04-30T06:26:26.609ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
37332025-04-30T06:26:26.613ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db00819f-e7fb-48c6-9df4-b76b769ab1ba", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37342025-04-30T06:26:26.613ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
37352025-04-30T06:26:26.616ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a40192c-cc0d-41c8-aa79-b34c316bd150", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37362025-04-30T06:26:26.616ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
37372025-04-30T06:26:26.618ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1942a14b-8db5-4c7c-b9ae-2914fcba24b7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37382025-04-30T06:26:26.618ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
37392025-04-30T06:26:26.623ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a3b2661-cf32-4921-89a3-4b7d2b2f0f76", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37402025-04-30T06:26:26.623ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
37412025-04-30T06:26:26.624ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
37422025-04-30T06:26:26.627ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06230f35-9e17-4f69-8d36-6aba89f8bb3c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37432025-04-30T06:26:26.628ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
37442025-04-30T06:26:26.628ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da7b8bbb-f35a-4944-aacf-1b7e814ef5fd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37452025-04-30T06:26:26.628ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
37462025-04-30T06:26:26.630ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c505d996-7e6f-4b55-802c-4725aa4a9a92", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37472025-04-30T06:26:26.630ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
37482025-04-30T06:26:26.632ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a63aa44-06be-4c14-9289-1fb64a8e926a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37492025-04-30T06:26:26.633ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
37502025-04-30T06:26:26.638ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29b3f89c-3e4a-4d68-ad47-5c61673dff38", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37512025-04-30T06:26:26.638ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
37522025-04-30T06:26:26.645ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "008224f8-1c97-4740-9c0b-4866f1401d78", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37532025-04-30T06:26:26.646ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
37542025-04-30T06:26:26.649ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff36a28f-3a35-4ec0-94e6-15765c2d2fae", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37552025-04-30T06:26:26.650ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
37562025-04-30T06:26:26.654ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c95cf02-55ad-4f56-840b-b6c4444989ee", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37572025-04-30T06:26:26.654ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
37582025-04-30T06:26:26.659ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a75497c8-15ad-4dba-a780-73e5680ca945", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37592025-04-30T06:26:26.659ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
37602025-04-30T06:26:26.663ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a945908a-fb55-4e87-876e-367ef2258800", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37612025-04-30T06:26:26.664ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
37622025-04-30T06:26:26.665ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e215952-e4e7-452a-a229-897720b54ab6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37632025-04-30T06:26:26.665ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
37642025-04-30T06:26:26.667ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70a3b8e9-93ef-457e-90e6-29065d4ca224", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37652025-04-30T06:26:26.667ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
37662025-04-30T06:26:26.669ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "258c8228-5f66-4301-beb4-5ba2b3fd1d76", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37672025-04-30T06:26:26.669ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
37682025-04-30T06:26:26.670ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a9554b9-d4d3-4f11-9d7f-3187c671a6c6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37692025-04-30T06:26:26.671ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
37702025-04-30T06:26:26.673ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "802bf15a-ac65-4174-b7c2-4a238ddff817", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37712025-04-30T06:26:26.673ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
37722025-04-30T06:26:26.674ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37732025-04-30T06:26:26.675ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37742025-04-30T06:26:26.675ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37752025-04-30T06:26:26.675ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
37762025-04-30T06:26:26.676ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37772025-04-30T06:26:26.679ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37782025-04-30T06:26:26.687ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37792025-04-30T06:26:26.687ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37802025-04-30T06:26:26.687ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
37812025-04-30T06:26:26.688ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37822025-04-30T06:26:26.689ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b27f8a0a-862a-4e4f-a507-6dec357afdf3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37832025-04-30T06:26:26.689ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
37842025-04-30T06:26:26.693ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37852025-04-30T06:26:26.693ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
37862025-04-30T06:26:26.694ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37872025-04-30T06:26:26.699ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bc810dd-50e4-4b78-b4d3-cd4b084c87c1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37882025-04-30T06:26:26.699ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
37892025-04-30T06:26:26.701ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "24a0f51f-59b8-45b6-9c37-8df5101f1ad2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37902025-04-30T06:26:26.702ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
37912025-04-30T06:26:26.705ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4d75359-afae-44ab-8688-58228dd78645", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37922025-04-30T06:26:26.705ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
37932025-04-30T06:26:26.720ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15d94bc9-e188-43e1-adaa-ebca9a451d2c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37942025-04-30T06:26:26.721ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
37952025-04-30T06:26:26.727ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05c07474-79d4-486b-9039-27f29b178d18", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37962025-04-30T06:26:26.728ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
37972025-04-30T06:26:26.729ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7efb70e-b514-440c-a98e-8fe9386356a0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
37982025-04-30T06:26:26.730ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
37992025-04-30T06:26:26.731ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d5ecf5d-61f1-476c-9cb0-16ac40efdbf1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38002025-04-30T06:26:26.731ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
38012025-04-30T06:26:26.733ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de995078-77c6-41d5-a684-baf80fad537a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38022025-04-30T06:26:26.733ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
38032025-04-30T06:26:26.735ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ef6a174-1688-4a50-b267-fd022eb4a4ed", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38042025-04-30T06:26:26.735ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
38052025-04-30T06:26:26.738ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
38062025-04-30T06:26:26.739ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3150cf2-0301-437b-81ff-6c24b995d7f0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38072025-04-30T06:26:26.740ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
38082025-04-30T06:26:26.743ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80357e11-7781-45ad-b78a-0d1b9556ab5d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38092025-04-30T06:26:26.743ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
38102025-04-30T06:26:26.747ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4387c2d9-3cb6-4aff-979a-d32a4c41cba6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38112025-04-30T06:26:26.747ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
38122025-04-30T06:26:26.750ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4f6e48b-ea65-494b-a0e5-cc4b1fdd31a9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38132025-04-30T06:26:26.750ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
38142025-04-30T06:26:26.754ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97c4c578-4cfa-4f67-b0b5-6e809d7b9f89", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38152025-04-30T06:26:26.754ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
38162025-04-30T06:26:26.758ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aac5963c-e307-48b1-abc8-4651309313f0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38172025-04-30T06:26:26.758ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
38182025-04-30T06:26:26.762ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a4b94d7-5863-46a4-b819-9a41c5e955ae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38192025-04-30T06:26:26.762ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
38202025-04-30T06:26:26.763ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "292ab300-712c-4ca1-b2f3-09293fb83b43", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38212025-04-30T06:26:26.763ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
38222025-04-30T06:26:26.764ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ca27cb0-a317-405a-a018-96ef62ac69c9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38232025-04-30T06:26:26.765ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
38242025-04-30T06:26:26.766ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c5e60e7-81b5-4a76-9406-9fa283e5ba37", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38252025-04-30T06:26:26.766ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
38262025-04-30T06:26:26.767ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cbb9949-45e3-497b-8ff1-0873cfff2d14", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38272025-04-30T06:26:26.768ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
38282025-04-30T06:26:26.768ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38292025-04-30T06:26:26.768ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38302025-04-30T06:26:26.768ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38312025-04-30T06:26:26.769ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38322025-04-30T06:26:26.772ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38332025-04-30T06:26:26.777ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
38342025-04-30T06:26:26.777ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
38352025-04-30T06:26:26.777ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38362025-04-30T06:26:26.777ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38372025-04-30T06:26:26.777ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38382025-04-30T06:26:26.777ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38392025-04-30T06:26:26.780ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38402025-04-30T06:26:26.786ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
38412025-04-30T06:26:26.786ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38422025-04-30T06:26:26.786ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38432025-04-30T06:26:26.787ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38442025-04-30T06:26:26.789ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38452025-04-30T06:26:26.789ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38462025-04-30T06:26:26.790ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38472025-04-30T06:26:26.790ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
38482025-04-30T06:26:26.790ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38492025-04-30T06:26:26.793ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38502025-04-30T06:26:26.806ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38512025-04-30T06:26:26.806ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38522025-04-30T06:26:26.806ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
38532025-04-30T06:26:26.806ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38542025-04-30T06:26:26.897ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
38552025-04-30T06:26:26.899ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
38562025-04-30T06:26:26.901ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:59009 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
38572025-04-30T06:26:26.901ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
38582025-04-30T06:26:26.902ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38592025-04-30T06:26:26.907ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = c5e8d94d-faec-46eb-b955-7d21a03f51a1 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38602025-04-30T06:26:26.907ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:59009 req_id = c8d65134-9b8f-427a-9108-268247b8b7b4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38612025-04-30T06:26:26.907ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = c5e8d94d-faec-46eb-b955-7d21a03f51a1 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38622025-04-30T06:26:26.907ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = c5e8d94d-faec-46eb-b955-7d21a03f51a1 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38632025-04-30T06:26:26.908ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = c5e8d94d-faec-46eb-b955-7d21a03f51a1 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38642025-04-30T06:26:26.908ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1569 local_addr = [::1]:44559 method = GET remote_addr = [::1]:59009 req_id = c8d65134-9b8f-427a-9108-268247b8b7b4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38652025-04-30T06:26:26.910ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38662025-04-30T06:26:26.910ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
38672025-04-30T06:26:26.910ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38682025-04-30T06:26:26.910ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8d65134-9b8f-427a-9108-268247b8b7b4", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38692025-04-30T06:26:26.912ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38702025-04-30T06:26:26.912ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38712025-04-30T06:26:26.912ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
38722025-04-30T06:26:26.914ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38732025-04-30T06:26:26.914ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38742025-04-30T06:26:26.914ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38752025-04-30T06:26:26.914ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38762025-04-30T06:26:26.914ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38772025-04-30T06:26:26.918ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 11066 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = c5e8d94d-faec-46eb-b955-7d21a03f51a1 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
38782025-04-30T06:26:26.919ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5e8d94d-faec-46eb-b955-7d21a03f51a1", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:25 GMT"} })
38792025-04-30T06:26:26.919ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38802025-04-30T06:26:26.922ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 26357413-9e0c-4b17-bc46-9b01c91e502a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38812025-04-30T06:26:26.923ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38822025-04-30T06:26:26.935ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38832025-04-30T06:26:26.935ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38842025-04-30T06:26:26.935ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38852025-04-30T06:26:26.935ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38862025-04-30T06:26:26.936ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 26357413-9e0c-4b17-bc46-9b01c91e502a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38872025-04-30T06:26:26.936ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 26357413-9e0c-4b17-bc46-9b01c91e502a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38882025-04-30T06:26:26.936ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 26357413-9e0c-4b17-bc46-9b01c91e502a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38892025-04-30T06:26:26.962ZTRACtest_disk_expunge (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38902025-04-30T06:26:26.962ZTRACtest_disk_expunge (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
38912025-04-30T06:26:26.963ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 40924 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 26357413-9e0c-4b17-bc46-9b01c91e502a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
38922025-04-30T06:26:26.966ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "26357413-9e0c-4b17-bc46-9b01c91e502a", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
38932025-04-30T06:26:26.966ZDEBGtest_disk_expunge (oximeter-agent): registered new metric producer address = 127.0.0.1:38728 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
38942025-04-30T06:26:27.037ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
38952025-04-30T06:26:27.041ZDEBGtest_disk_expunge (oximeter-agent): registered new metric producer address = [::1]:43945 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
38962025-04-30T06:26:27.107ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
38972025-04-30T06:26:27.111ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
38982025-04-30T06:26:27.151ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
38992025-04-30T06:26:27.209ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39002025-04-30T06:26:27.209ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39012025-04-30T06:26:27.209ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39022025-04-30T06:26:27.209ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
39032025-04-30T06:26:27.210ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39042025-04-30T06:26:27.214ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39052025-04-30T06:26:27.225ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39062025-04-30T06:26:27.225ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39072025-04-30T06:26:27.225ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
39082025-04-30T06:26:27.226ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39092025-04-30T06:26:27.227ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
39102025-04-30T06:26:27.227ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
39112025-04-30T06:26:27.228ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 767.627507ms iteration = 6 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
39122025-04-30T06:26:27.228ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 7 reason = Signaled
39132025-04-30T06:26:27.228ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
39142025-04-30T06:26:27.228ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39152025-04-30T06:26:27.228ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39162025-04-30T06:26:27.228ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39172025-04-30T06:26:27.229ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39182025-04-30T06:26:27.231ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39192025-04-30T06:26:27.232ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
39202025-04-30T06:26:27.232ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39212025-04-30T06:26:27.233ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39222025-04-30T06:26:27.238ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
39232025-04-30T06:26:27.238ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39242025-04-30T06:26:27.239ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39252025-04-30T06:26:27.239ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39262025-04-30T06:26:27.247ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
39272025-04-30T06:26:27.247ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
39282025-04-30T06:26:27.247ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b3 01 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61791 req_id = db54d1a9-e1fe-44fd-85da-7d70738abf56
39292025-04-30T06:26:27.248ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45825,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61791 req_id = db54d1a9-e1fe-44fd-85da-7d70738abf56
39302025-04-30T06:26:27.249ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
39312025-04-30T06:26:27.250ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
39322025-04-30T06:26:27.250ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61791 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = db54d1a9-e1fe-44fd-85da-7d70738abf56
39332025-04-30T06:26:27.252ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3252, tv_nsec: 51267702 } })
39342025-04-30T06:26:27.252ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
39352025-04-30T06:26:27.252ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 55 49 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e UI..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65450 req_id = 34eb6892-1b7e-402b-9412-9e6a9be74994
39362025-04-30T06:26:27.252ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21833,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65450 req_id = 34eb6892-1b7e-402b-9412-9e6a9be74994
39372025-04-30T06:26:27.253ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
39382025-04-30T06:26:27.253ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65450 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 34eb6892-1b7e-402b-9412-9e6a9be74994
39392025-04-30T06:26:27.305ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
39402025-04-30T06:26:27.306ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: af 6c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .l..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54209 req_id = e62ebf20-c625-4912-96d9-0ca50139fa17
39412025-04-30T06:26:27.307ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44908,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54209 req_id = e62ebf20-c625-4912-96d9-0ca50139fa17
39422025-04-30T06:26:27.307ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
39432025-04-30T06:26:27.308ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44908, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54209 req_id = e62ebf20-c625-4912-96d9-0ca50139fa17
39442025-04-30T06:26:27.308ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
39452025-04-30T06:26:27.308ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
39462025-04-30T06:26:27.308ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
39472025-04-30T06:26:27.311ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3c06c65-3172-477e-aaf8-78ac4aa3028f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39482025-04-30T06:26:27.312ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
39492025-04-30T06:26:27.315ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc84da8f-3244-46fa-8f56-8f5c8f40d6c0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39502025-04-30T06:26:27.315ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
39512025-04-30T06:26:27.320ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82cfcee5-bb4c-43f3-9c8f-364ae0799118", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39522025-04-30T06:26:27.321ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
39532025-04-30T06:26:27.323ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74b76f24-a042-441e-9e25-0c809c421aea", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39542025-04-30T06:26:27.323ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
39552025-04-30T06:26:27.325ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7920c3d-8589-4de0-b676-1a38b070a442", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39562025-04-30T06:26:27.325ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
39572025-04-30T06:26:27.328ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99e07d8d-d5a7-4d51-b8ea-0605066d7eb8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39582025-04-30T06:26:27.328ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
39592025-04-30T06:26:27.333ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
39602025-04-30T06:26:27.334ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ca89c07-b8ef-4700-9a79-a38002d37f37", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39612025-04-30T06:26:27.334ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
39622025-04-30T06:26:27.336ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fe815d0-5b3f-4887-a896-04c876b11f49", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39632025-04-30T06:26:27.336ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
39642025-04-30T06:26:27.339ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc02d1e9-e652-4752-820c-14e4ba5ce6fa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39652025-04-30T06:26:27.339ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
39662025-04-30T06:26:27.341ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfb7b9fb-e901-4ac0-96c5-1644f7ffb34f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39672025-04-30T06:26:27.341ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
39682025-04-30T06:26:27.343ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b2f7613-f18b-465c-a270-11f10d45c347", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39692025-04-30T06:26:27.343ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
39702025-04-30T06:26:27.349ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96cf3202-ea45-49d3-b2eb-2f4ba08b1f6e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39712025-04-30T06:26:27.349ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
39722025-04-30T06:26:27.351ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4398049-6d0c-4a81-bff4-4723ca9659f3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39732025-04-30T06:26:27.352ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
39742025-04-30T06:26:27.355ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb886266-825b-43ef-a303-7598a03d2858", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39752025-04-30T06:26:27.355ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
39762025-04-30T06:26:27.357ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34541670-e5d3-43f9-bb75-c53a36b537c8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39772025-04-30T06:26:27.357ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
39782025-04-30T06:26:27.360ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "691afab1-e3e6-4e42-88de-e4c79199f17f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39792025-04-30T06:26:27.360ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
39802025-04-30T06:26:27.362ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ee602b-53ea-451d-a9f7-41c80351d061", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39812025-04-30T06:26:27.362ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
39822025-04-30T06:26:27.364ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a72baffc-b3ba-4592-a006-7178d17792a2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39832025-04-30T06:26:27.364ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
39842025-04-30T06:26:27.367ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd5faa08-503b-4b87-a2ac-524aaeaeb945", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39852025-04-30T06:26:27.367ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
39862025-04-30T06:26:27.369ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1305f69-bcf4-4d59-8ebb-c1432d8edd6b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39872025-04-30T06:26:27.369ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
39882025-04-30T06:26:27.370ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1165fd5-9092-47c8-8f2e-17a3b744be05", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39892025-04-30T06:26:27.370ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
39902025-04-30T06:26:27.371ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a098d7c-3e78-4dd7-a540-8ebfe91d5c85", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39912025-04-30T06:26:27.371ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
39922025-04-30T06:26:27.372ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c68d94ca-f6b9-4cb7-8576-410969f472f6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39932025-04-30T06:26:27.372ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
39942025-04-30T06:26:27.374ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ffd3d22-d5e6-4ab6-99cb-46dccf04a241", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39952025-04-30T06:26:27.374ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
39962025-04-30T06:26:27.381ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c37fd392-ce3f-4eb8-a1b5-f609d35b7bf9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
39972025-04-30T06:26:27.381ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
39982025-04-30T06:26:27.384ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39992025-04-30T06:26:27.384ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40002025-04-30T06:26:27.384ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40012025-04-30T06:26:27.384ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
40022025-04-30T06:26:27.385ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40032025-04-30T06:26:27.387ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94988b94-aa8c-416c-ad3c-92b15e02a78d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40042025-04-30T06:26:27.387ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
40052025-04-30T06:26:27.388ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40062025-04-30T06:26:27.394ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40072025-04-30T06:26:27.394ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40082025-04-30T06:26:27.394ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
40092025-04-30T06:26:27.395ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40102025-04-30T06:26:27.402ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40112025-04-30T06:26:27.402ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
40122025-04-30T06:26:27.403ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40132025-04-30T06:26:27.404ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6bc951d-c3f2-4c52-b0b6-c94abde4ef03", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40142025-04-30T06:26:27.404ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
40152025-04-30T06:26:27.408ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df474fda-663a-4039-9cac-99fdb2052ccb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40162025-04-30T06:26:27.409ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
40172025-04-30T06:26:27.413ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "389bb216-5e75-4ec5-8e32-3ebbd019ef69", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40182025-04-30T06:26:27.413ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
40192025-04-30T06:26:27.416ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5be4f73-ed9f-4013-a9da-819f8688d481", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40202025-04-30T06:26:27.416ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
40212025-04-30T06:26:27.417ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "612d98f2-3f29-4ec6-87b8-fc350dbfc5ea", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40222025-04-30T06:26:27.417ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
40232025-04-30T06:26:27.419ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2bd09b2-d5d8-487d-be75-623079fd2ceb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40242025-04-30T06:26:27.419ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
40252025-04-30T06:26:27.422ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f08769cc-77c8-4965-a7f4-f481f078f552", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40262025-04-30T06:26:27.423ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
40272025-04-30T06:26:27.424ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f2931ff-1e40-435e-af11-fd88d4ab3147", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40282025-04-30T06:26:27.424ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
40292025-04-30T06:26:27.426ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "024713bd-e08f-482f-a199-6c198e658b7b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40302025-04-30T06:26:27.426ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
40312025-04-30T06:26:27.428ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e38cfe4-04d1-47e0-a747-8c2d8559a83a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40322025-04-30T06:26:27.428ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
40332025-04-30T06:26:27.431ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70ba5f97-d25e-4169-a4de-5bff7be706f8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40342025-04-30T06:26:27.431ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
40352025-04-30T06:26:27.437ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71b4d92d-80eb-4bd3-8f0d-ea0d03fd0dcb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40362025-04-30T06:26:27.438ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
40372025-04-30T06:26:27.438ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
40382025-04-30T06:26:27.442ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cea8cd2-80ef-47d8-81e5-e8feee46c5ec", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40392025-04-30T06:26:27.442ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
40402025-04-30T06:26:27.445ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33be818a-aaf3-4195-9ef7-61947db45046", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40412025-04-30T06:26:27.445ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
40422025-04-30T06:26:27.448ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4866724-b7b6-45c8-b956-08446694f50c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40432025-04-30T06:26:27.448ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
40442025-04-30T06:26:27.450ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "847c66b7-efa4-4323-8d44-1d18714c829b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40452025-04-30T06:26:27.450ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
40462025-04-30T06:26:27.451ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a28a4324-020a-44fd-a8a8-9c87a3fd215d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40472025-04-30T06:26:27.451ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
40482025-04-30T06:26:27.453ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bd3a0c2-1b4c-403f-84e6-0a6dfb09fab2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40492025-04-30T06:26:27.453ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
40502025-04-30T06:26:27.455ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "860fa195-5693-4b43-be9d-3f3d9e9bf20c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40512025-04-30T06:26:27.455ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40522025-04-30T06:26:27.455ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40532025-04-30T06:26:27.455ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40542025-04-30T06:26:27.455ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40552025-04-30T06:26:27.456ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40562025-04-30T06:26:27.460ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40572025-04-30T06:26:27.467ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40582025-04-30T06:26:27.467ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40592025-04-30T06:26:27.467ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40602025-04-30T06:26:27.467ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40612025-04-30T06:26:27.467ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40622025-04-30T06:26:27.469ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40632025-04-30T06:26:27.471ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40642025-04-30T06:26:27.478ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40652025-04-30T06:26:27.478ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40662025-04-30T06:26:27.478ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40672025-04-30T06:26:27.479ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40682025-04-30T06:26:27.545ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
40692025-04-30T06:26:27.545ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
40702025-04-30T06:26:27.546ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40712025-04-30T06:26:27.546ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40722025-04-30T06:26:27.546ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40732025-04-30T06:26:27.547ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
40742025-04-30T06:26:27.547ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40752025-04-30T06:26:27.547ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:62203 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
40762025-04-30T06:26:27.548ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:62203 req_id = 000d2aee-40f7-4d64-8718-bb7ee0d38341 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40772025-04-30T06:26:27.549ZINFOtest_disk_expunge (dropshot): request completed latency_us = 567 local_addr = [::1]:44559 method = GET remote_addr = [::1]:62203 req_id = 000d2aee-40f7-4d64-8718-bb7ee0d38341 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40782025-04-30T06:26:27.549ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "000d2aee-40f7-4d64-8718-bb7ee0d38341", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
40792025-04-30T06:26:27.551ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40802025-04-30T06:26:27.552ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40812025-04-30T06:26:27.552ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
40822025-04-30T06:26:27.561ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40832025-04-30T06:26:27.561ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40842025-04-30T06:26:27.561ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40852025-04-30T06:26:27.561ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40862025-04-30T06:26:27.563ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40872025-04-30T06:26:27.564ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40882025-04-30T06:26:27.577ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40892025-04-30T06:26:27.577ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40902025-04-30T06:26:27.577ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
40912025-04-30T06:26:27.577ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40922025-04-30T06:26:27.583ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40932025-04-30T06:26:27.591ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40942025-04-30T06:26:27.591ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40952025-04-30T06:26:27.591ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40962025-04-30T06:26:27.592ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40972025-04-30T06:26:27.596ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40982025-04-30T06:26:27.596ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
40992025-04-30T06:26:27.597ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41002025-04-30T06:26:27.675ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
41012025-04-30T06:26:27.675ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
41022025-04-30T06:26:27.675ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 447.101932ms iteration = 7 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
41032025-04-30T06:26:27.675ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 8 reason = Signaled
41042025-04-30T06:26:27.675ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
41052025-04-30T06:26:27.675ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41062025-04-30T06:26:27.675ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41072025-04-30T06:26:27.675ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41082025-04-30T06:26:27.676ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41092025-04-30T06:26:27.676ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
41102025-04-30T06:26:27.710ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41112025-04-30T06:26:27.715ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
41122025-04-30T06:26:27.716ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41132025-04-30T06:26:27.716ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41142025-04-30T06:26:27.717ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41152025-04-30T06:26:27.726ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
41162025-04-30T06:26:27.739ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
41172025-04-30T06:26:27.742ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41182025-04-30T06:26:27.742ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41192025-04-30T06:26:27.742ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41202025-04-30T06:26:27.742ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
41212025-04-30T06:26:27.743ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41222025-04-30T06:26:27.744ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e2 5b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .[..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53240 req_id = 096865df-b4a7-476e-9282-4bd122731f33
41232025-04-30T06:26:27.744ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57947,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53240 req_id = 096865df-b4a7-476e-9282-4bd122731f33
41242025-04-30T06:26:27.745ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
41252025-04-30T06:26:27.745ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
41262025-04-30T06:26:27.745ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53240 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 096865df-b4a7-476e-9282-4bd122731f33
41272025-04-30T06:26:27.747ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3252, tv_nsec: 546369951 } })
41282025-04-30T06:26:27.747ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
41292025-04-30T06:26:27.748ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41302025-04-30T06:26:27.760ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41312025-04-30T06:26:27.760ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41322025-04-30T06:26:27.760ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
41332025-04-30T06:26:27.760ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41342025-04-30T06:26:27.761ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d7 db 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60511 req_id = 1ae5a278-1d1e-4e3f-b880-ad06f5698045
41352025-04-30T06:26:27.762ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55259,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60511 req_id = 1ae5a278-1d1e-4e3f-b880-ad06f5698045
41362025-04-30T06:26:27.762ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
41372025-04-30T06:26:27.762ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60511 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1ae5a278-1d1e-4e3f-b880-ad06f5698045
41382025-04-30T06:26:27.808ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
41392025-04-30T06:26:27.809ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41402025-04-30T06:26:27.809ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
41412025-04-30T06:26:27.809ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41422025-04-30T06:26:27.810ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3b c1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ;............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53031 req_id = 03aed241-9200-40cb-8287-1c3c0619dd58
41432025-04-30T06:26:27.810ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15297,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53031 req_id = 03aed241-9200-40cb-8287-1c3c0619dd58
41442025-04-30T06:26:27.810ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
41452025-04-30T06:26:27.810ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15297, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53031 req_id = 03aed241-9200-40cb-8287-1c3c0619dd58
41462025-04-30T06:26:27.810ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
41472025-04-30T06:26:27.810ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
41482025-04-30T06:26:27.810ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
41492025-04-30T06:26:27.813ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1501b9ba-d7fe-4d32-be5a-323cdc877770", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41502025-04-30T06:26:27.814ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
41512025-04-30T06:26:27.815ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3f840f8-5063-4f39-a012-259f742fb1c0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41522025-04-30T06:26:27.815ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
41532025-04-30T06:26:27.819ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7951ab9d-4542-420d-af7b-e2777af42700", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41542025-04-30T06:26:27.819ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
41552025-04-30T06:26:27.821ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a728f528-ac28-4ca4-8d31-a2a663693414", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41562025-04-30T06:26:27.821ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
41572025-04-30T06:26:27.824ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d02ebf50-89db-4061-a93a-343e3e66352d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41582025-04-30T06:26:27.824ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
41592025-04-30T06:26:27.828ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfd004e0-6f6e-49f4-a87e-a8b03518bf78", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41602025-04-30T06:26:27.829ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
41612025-04-30T06:26:27.836ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e795fb5e-e18d-4e51-bc7b-23ccb13be398", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41622025-04-30T06:26:27.837ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
41632025-04-30T06:26:27.842ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef64f347-1bb9-4fad-bcaf-7292824a7da5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41642025-04-30T06:26:27.842ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
41652025-04-30T06:26:27.844ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2146e564-17bb-4816-993d-884ce9453dc0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41662025-04-30T06:26:27.844ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
41672025-04-30T06:26:27.845ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c657409-baa7-4da3-9d85-3fad63e98cda", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41682025-04-30T06:26:27.845ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
41692025-04-30T06:26:27.848ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1857e6e9-612f-4a59-b06f-f050ebb1a956", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41702025-04-30T06:26:27.863ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
41712025-04-30T06:26:27.863ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
41722025-04-30T06:26:27.865ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "431a61c5-51b3-4f87-b80e-b9281b1516d2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41732025-04-30T06:26:27.865ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
41742025-04-30T06:26:27.867ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9d6ad9c-06af-418d-bb60-343161d8d328", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41752025-04-30T06:26:27.867ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
41762025-04-30T06:26:27.869ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "244110ec-91d0-4359-ba48-05f65635c58a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41772025-04-30T06:26:27.869ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
41782025-04-30T06:26:27.871ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1df62a1f-9d3c-4f19-bfe5-ab0a7d2a94b2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41792025-04-30T06:26:27.871ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
41802025-04-30T06:26:27.874ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48702af8-988a-4538-a228-bbe1102d7872", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41812025-04-30T06:26:27.874ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
41822025-04-30T06:26:27.876ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37ea5160-d1a6-41ca-9371-23aa510414ee", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41832025-04-30T06:26:27.877ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
41842025-04-30T06:26:27.879ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ceb21db-2f6e-4e29-ac17-fff537a2881e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41852025-04-30T06:26:27.880ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
41862025-04-30T06:26:27.881ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d42d10c-e84e-4a93-8aa4-f016777ff60a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41872025-04-30T06:26:27.881ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
41882025-04-30T06:26:27.883ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2690b6b-06e2-4c31-a7ba-e5d827e422d4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41892025-04-30T06:26:27.883ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
41902025-04-30T06:26:27.884ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca85538f-0ea6-4a29-b702-2b8c95e0532e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41912025-04-30T06:26:27.884ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
41922025-04-30T06:26:27.886ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "007509c3-7849-42e2-a7b1-2d1efca0dd06", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41932025-04-30T06:26:27.886ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
41942025-04-30T06:26:27.888ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "363cf80b-5706-4074-899e-486ab60582d2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41952025-04-30T06:26:27.889ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
41962025-04-30T06:26:27.893ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38d8051b-32f8-4ba2-aa93-333dba03b827", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41972025-04-30T06:26:27.893ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
41982025-04-30T06:26:27.894ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9d83c50-2aab-4ebf-bcec-49a170f5f91b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
41992025-04-30T06:26:27.894ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
42002025-04-30T06:26:27.898ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de5f9233-33c8-418b-bda9-6d39ce7417ee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42012025-04-30T06:26:27.898ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
42022025-04-30T06:26:27.899ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "228c04ba-c4b3-43a0-9225-a21d600b9cd2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42032025-04-30T06:26:27.899ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
42042025-04-30T06:26:27.901ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddfeb98c-24e3-4090-bdc4-f80520a8b456", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42052025-04-30T06:26:27.901ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
42062025-04-30T06:26:27.903ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fa3d8ed-5769-4907-a7db-356ae302d5ef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42072025-04-30T06:26:27.903ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
42082025-04-30T06:26:27.908ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1ef314b-f8c2-4afb-b133-b7af42dca686", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42092025-04-30T06:26:27.908ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
42102025-04-30T06:26:27.909ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "645b2b8f-308a-4dab-9aef-f8d105fb9556", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42112025-04-30T06:26:27.909ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
42122025-04-30T06:26:27.910ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32da4d18-dace-4137-af64-9e2179d20e55", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42132025-04-30T06:26:27.911ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
42142025-04-30T06:26:27.912ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "372b877b-2826-4a17-ba58-474ca37f90a7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42152025-04-30T06:26:27.912ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
42162025-04-30T06:26:27.914ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42172025-04-30T06:26:27.914ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42182025-04-30T06:26:27.914ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42192025-04-30T06:26:27.914ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
42202025-04-30T06:26:27.915ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42212025-04-30T06:26:27.916ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73fded1e-20a8-4302-91f6-9c468b7b8429", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42222025-04-30T06:26:27.916ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
42232025-04-30T06:26:27.917ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3fdae0a-08cf-41fe-95de-c22c9bcfeac4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42242025-04-30T06:26:27.918ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
42252025-04-30T06:26:27.918ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42262025-04-30T06:26:27.924ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42272025-04-30T06:26:27.924ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42282025-04-30T06:26:27.924ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
42292025-04-30T06:26:27.925ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42302025-04-30T06:26:27.929ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42312025-04-30T06:26:27.929ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
42322025-04-30T06:26:27.930ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42332025-04-30T06:26:27.930ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ff769fe-1146-4e39-b100-1bbf6cb7893b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42342025-04-30T06:26:27.931ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
42352025-04-30T06:26:27.938ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a74341fe-e2fe-4b3f-b47a-ec10bcaecd4e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42362025-04-30T06:26:27.938ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
42372025-04-30T06:26:27.941ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "324085ed-7f65-40eb-b016-c6e5bb299ba2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42382025-04-30T06:26:27.941ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
42392025-04-30T06:26:27.944ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fa969be-8d70-49d0-8699-ff12fb89a770", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42402025-04-30T06:26:27.944ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
42412025-04-30T06:26:27.948ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66668413-d6b6-4ec7-a16d-360539e6f691", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42422025-04-30T06:26:27.948ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
42432025-04-30T06:26:27.952ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a9c44d9-8db6-4988-90ad-d16ee2a6a41d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42442025-04-30T06:26:27.952ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
42452025-04-30T06:26:27.953ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5822649b-4bbc-4340-ae72-8c5ece7d97b6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42462025-04-30T06:26:27.954ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
42472025-04-30T06:26:27.954ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "baacfe17-f7b8-4a04-8b44-5d8596b67d93", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42482025-04-30T06:26:27.955ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
42492025-04-30T06:26:27.956ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3509df36-1c87-4dec-9a79-a73612944b8a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42502025-04-30T06:26:27.956ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
42512025-04-30T06:26:27.956ZTRACtest_disk_expunge (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42522025-04-30T06:26:27.956ZTRACtest_disk_expunge (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
42532025-04-30T06:26:27.957ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ba0d6db-6ce6-44f1-92e6-ced022a2d0ba", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:26 GMT"} })
42542025-04-30T06:26:27.957ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42552025-04-30T06:26:27.957ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42562025-04-30T06:26:27.957ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42572025-04-30T06:26:27.957ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42582025-04-30T06:26:27.959ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42592025-04-30T06:26:27.967ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
42602025-04-30T06:26:27.967ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42612025-04-30T06:26:27.975ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42622025-04-30T06:26:27.975ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42632025-04-30T06:26:27.975ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42642025-04-30T06:26:27.975ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42652025-04-30T06:26:27.975ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42662025-04-30T06:26:27.976ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42672025-04-30T06:26:27.984ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42682025-04-30T06:26:27.989ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42692025-04-30T06:26:27.989ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42702025-04-30T06:26:27.989ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42712025-04-30T06:26:27.990ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42722025-04-30T06:26:28.059ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
42732025-04-30T06:26:28.059ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
42742025-04-30T06:26:28.060ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42752025-04-30T06:26:28.060ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42762025-04-30T06:26:28.060ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42772025-04-30T06:26:28.060ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
42782025-04-30T06:26:28.061ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42792025-04-30T06:26:28.061ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:41717 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
42802025-04-30T06:26:28.061ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
42812025-04-30T06:26:28.061ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
42822025-04-30T06:26:28.066ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:41717 req_id = 82ec67d1-bfa0-4fec-9397-667257a2db16 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42832025-04-30T06:26:28.067ZINFOtest_disk_expunge (dropshot): request completed latency_us = 886 local_addr = [::1]:44559 method = GET remote_addr = [::1]:41717 req_id = 82ec67d1-bfa0-4fec-9397-667257a2db16 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42842025-04-30T06:26:28.067ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42852025-04-30T06:26:28.076ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42862025-04-30T06:26:28.076ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42872025-04-30T06:26:28.076ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
42882025-04-30T06:26:28.077ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42892025-04-30T06:26:28.078ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "82ec67d1-bfa0-4fec-9397-667257a2db16", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
42902025-04-30T06:26:28.079ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42912025-04-30T06:26:28.080ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42922025-04-30T06:26:28.080ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
42932025-04-30T06:26:28.081ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42942025-04-30T06:26:28.081ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42952025-04-30T06:26:28.081ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42962025-04-30T06:26:28.081ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42972025-04-30T06:26:28.082ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42982025-04-30T06:26:28.087ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
42992025-04-30T06:26:28.088ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43002025-04-30T06:26:28.088ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43012025-04-30T06:26:28.088ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43022025-04-30T06:26:28.089ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43032025-04-30T06:26:28.090ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
43042025-04-30T06:26:28.090ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43052025-04-30T06:26:28.091ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43062025-04-30T06:26:28.096ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
43072025-04-30T06:26:28.097ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43082025-04-30T06:26:28.097ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43092025-04-30T06:26:28.097ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43102025-04-30T06:26:28.152ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
43112025-04-30T06:26:28.153ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
43122025-04-30T06:26:28.153ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 477.975384ms iteration = 8 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
43132025-04-30T06:26:28.153ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 9 reason = Signaled
43142025-04-30T06:26:28.153ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
43152025-04-30T06:26:28.153ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43162025-04-30T06:26:28.153ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43172025-04-30T06:26:28.153ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43182025-04-30T06:26:28.154ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43192025-04-30T06:26:28.166ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
43202025-04-30T06:26:28.168ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43212025-04-30T06:26:28.172ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
43222025-04-30T06:26:28.172ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43232025-04-30T06:26:28.172ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43242025-04-30T06:26:28.172ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43252025-04-30T06:26:28.176ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
43262025-04-30T06:26:28.176ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
43272025-04-30T06:26:28.177ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 58 12 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 X............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62112 req_id = 4d8e458a-41b0-4f67-bc99-b2c681212137
43282025-04-30T06:26:28.177ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22546,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62112 req_id = 4d8e458a-41b0-4f67-bc99-b2c681212137
43292025-04-30T06:26:28.177ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
43302025-04-30T06:26:28.177ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
43312025-04-30T06:26:28.177ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62112 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4d8e458a-41b0-4f67-bc99-b2c681212137
43322025-04-30T06:26:28.178ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3252, tv_nsec: 976957548 } })
43332025-04-30T06:26:28.178ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
43342025-04-30T06:26:28.178ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bd a2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56896 req_id = 5a2db8db-5a58-4501-a4f9-4b0ec9a82daf
43352025-04-30T06:26:28.178ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48546,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56896 req_id = 5a2db8db-5a58-4501-a4f9-4b0ec9a82daf
43362025-04-30T06:26:28.178ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
43372025-04-30T06:26:28.178ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56896 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5a2db8db-5a58-4501-a4f9-4b0ec9a82daf
43382025-04-30T06:26:28.225ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
43392025-04-30T06:26:28.226ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43402025-04-30T06:26:28.226ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43412025-04-30T06:26:28.226ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43422025-04-30T06:26:28.226ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
43432025-04-30T06:26:28.227ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43442025-04-30T06:26:28.228ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 48 67 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c Hg..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49674 req_id = a348c022-b7d7-4fcb-a75b-2667816e14f9
43452025-04-30T06:26:28.228ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18535,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49674 req_id = a348c022-b7d7-4fcb-a75b-2667816e14f9
43462025-04-30T06:26:28.228ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
43472025-04-30T06:26:28.228ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18535, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49674 req_id = a348c022-b7d7-4fcb-a75b-2667816e14f9
43482025-04-30T06:26:28.228ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
43492025-04-30T06:26:28.228ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
43502025-04-30T06:26:28.229ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
43512025-04-30T06:26:28.231ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43522025-04-30T06:26:28.239ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43532025-04-30T06:26:28.239ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43542025-04-30T06:26:28.240ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
43552025-04-30T06:26:28.240ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43562025-04-30T06:26:28.242ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "df7f9195-a2fa-439a-ac21-a6c0212591c2", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43572025-04-30T06:26:28.243ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
43582025-04-30T06:26:28.243ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43592025-04-30T06:26:28.243ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
43602025-04-30T06:26:28.244ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43612025-04-30T06:26:28.246ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18bd6c2e-bd79-451d-9d94-8b9bdc45ac20", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43622025-04-30T06:26:28.246ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
43632025-04-30T06:26:28.252ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2e38b76-728a-4bbe-a735-00acea293082", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43642025-04-30T06:26:28.252ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
43652025-04-30T06:26:28.257ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "450d692a-8981-472f-97bb-fd429d3f527b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43662025-04-30T06:26:28.257ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
43672025-04-30T06:26:28.275ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1ab3d52-5732-40a3-8960-35370e6302ab", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43682025-04-30T06:26:28.275ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
43692025-04-30T06:26:28.280ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca5652d7-b34b-4bac-b60f-fdcf656e9b02", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43702025-04-30T06:26:28.280ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
43712025-04-30T06:26:28.285ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4120cb41-b59b-4332-9db8-c709bd101c94", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43722025-04-30T06:26:28.286ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
43732025-04-30T06:26:28.291ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
43742025-04-30T06:26:28.297ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fe21341-e841-47cf-9bce-df96bd200b50", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43752025-04-30T06:26:28.297ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
43762025-04-30T06:26:28.298ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a536ae9-337c-4f65-a499-b322d78066ce", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43772025-04-30T06:26:28.298ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
43782025-04-30T06:26:28.299ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4af666d8-9f1f-4288-851b-490f18fa5503", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43792025-04-30T06:26:28.300ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
43802025-04-30T06:26:28.300ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbcf0b86-126a-40aa-b0c3-25b7249a184f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43812025-04-30T06:26:28.301ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
43822025-04-30T06:26:28.302ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a98eeb3-7aa0-4fcc-a005-faa482553e51", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43832025-04-30T06:26:28.302ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
43842025-04-30T06:26:28.304ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bab0be48-5020-49a8-bac1-ff236ec720c9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43852025-04-30T06:26:28.304ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
43862025-04-30T06:26:28.306ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df4b7bfe-7ce6-48fd-9355-0c8807fe5a91", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43872025-04-30T06:26:28.306ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
43882025-04-30T06:26:28.309ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56c83825-084f-4e36-b410-959f065e3854", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43892025-04-30T06:26:28.309ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
43902025-04-30T06:26:28.311ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aadfc29-095a-4b41-83ed-998bfae2cc2a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43912025-04-30T06:26:28.311ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
43922025-04-30T06:26:28.312ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7375589a-2372-479b-b53b-27a38729e164", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43932025-04-30T06:26:28.313ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
43942025-04-30T06:26:28.314ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d0bb4f6-e8bc-4ade-9bb1-d09073211b07", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43952025-04-30T06:26:28.315ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
43962025-04-30T06:26:28.316ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8853797f-c551-46bf-8dbb-8bf518d62c4f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43972025-04-30T06:26:28.316ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
43982025-04-30T06:26:28.317ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b8c742f-8bf6-42d0-aa18-cfde9d74a006", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
43992025-04-30T06:26:28.317ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
44002025-04-30T06:26:28.318ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc049734-c07d-4991-9c18-9f29e409839e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44012025-04-30T06:26:28.318ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
44022025-04-30T06:26:28.319ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "023e3056-a5aa-460d-8f06-cd9b6aa1a3dc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44032025-04-30T06:26:28.319ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
44042025-04-30T06:26:28.321ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7766c38b-8db3-4262-8d09-32208106f3ad", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44052025-04-30T06:26:28.321ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
44062025-04-30T06:26:28.322ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42a6b6a0-c242-4353-b11d-62ff872ec63e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44072025-04-30T06:26:28.322ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
44082025-04-30T06:26:28.324ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43a6761f-9e51-4a67-a858-95663203e3e6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44092025-04-30T06:26:28.324ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
44102025-04-30T06:26:28.327ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3aef469a-6908-477b-8607-c0f35cf5f69c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44112025-04-30T06:26:28.327ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
44122025-04-30T06:26:28.329ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d338d71f-d0b2-453a-96db-2c1a9209c113", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44132025-04-30T06:26:28.329ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
44142025-04-30T06:26:28.334ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95e75bf9-3b49-4103-86b5-760c428576df", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44152025-04-30T06:26:28.334ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
44162025-04-30T06:26:28.334ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10e1d8c8-d14e-4275-b305-5ad392258b9b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44172025-04-30T06:26:28.334ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
44182025-04-30T06:26:28.337ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a847236-b4c8-421e-93cb-d5bdb393ca42", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44192025-04-30T06:26:28.337ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
44202025-04-30T06:26:28.339ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1df0e06-c5c8-4aa2-b747-c2adcff6a564", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44212025-04-30T06:26:28.339ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
44222025-04-30T06:26:28.340ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f7d4845-737b-45b6-b704-c22f39d984c4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44232025-04-30T06:26:28.340ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
44242025-04-30T06:26:28.341ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c0d3e33-7d75-4b38-9cb8-7ddbdf001ab5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44252025-04-30T06:26:28.341ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
44262025-04-30T06:26:28.342ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44272025-04-30T06:26:28.343ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44282025-04-30T06:26:28.343ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44292025-04-30T06:26:28.343ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
44302025-04-30T06:26:28.344ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44312025-04-30T06:26:28.344ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "707b63df-bc4a-4cc0-922b-417ec0752f16", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44322025-04-30T06:26:28.344ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
44332025-04-30T06:26:28.345ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7396d79e-0377-4688-9684-703fc4def07c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44342025-04-30T06:26:28.346ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
44352025-04-30T06:26:28.347ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44362025-04-30T06:26:28.360ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44372025-04-30T06:26:28.360ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44382025-04-30T06:26:28.360ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
44392025-04-30T06:26:28.361ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44402025-04-30T06:26:28.368ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9fd5344-d958-45c1-a62e-165a04fffe7e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44412025-04-30T06:26:28.368ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
44422025-04-30T06:26:28.368ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44432025-04-30T06:26:28.368ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
44442025-04-30T06:26:28.369ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44452025-04-30T06:26:28.374ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6a8c381-118a-4391-b2cb-bb4e2008debc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44462025-04-30T06:26:28.374ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
44472025-04-30T06:26:28.381ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "efac5782-981e-49e8-ad07-f4736f03b877", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44482025-04-30T06:26:28.381ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
44492025-04-30T06:26:28.386ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "238438fb-a316-4881-aad6-eeba78dbc191", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44502025-04-30T06:26:28.387ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
44512025-04-30T06:26:28.392ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "114cda5a-d1cf-4941-86cc-cecb7d27078c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44522025-04-30T06:26:28.392ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
44532025-04-30T06:26:28.399ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
44542025-04-30T06:26:28.406ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba9668ae-befd-460b-a7e3-b2ab37f39908", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44552025-04-30T06:26:28.406ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
44562025-04-30T06:26:28.407ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2078ecd8-7541-4352-8ef4-d121c51017ac", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44572025-04-30T06:26:28.407ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
44582025-04-30T06:26:28.408ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ee446af-a057-4202-bb1f-f2cead13ca0b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44592025-04-30T06:26:28.408ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
44602025-04-30T06:26:28.410ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b17e35b-fdab-4917-a978-366dec63cf37", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44612025-04-30T06:26:28.410ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
44622025-04-30T06:26:28.411ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b2d9b66-e28f-4f5e-a734-b70491bfadc1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44632025-04-30T06:26:28.411ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44642025-04-30T06:26:28.411ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44652025-04-30T06:26:28.411ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44662025-04-30T06:26:28.411ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44672025-04-30T06:26:28.412ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44682025-04-30T06:26:28.415ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44692025-04-30T06:26:28.420ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44702025-04-30T06:26:28.420ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44712025-04-30T06:26:28.420ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44722025-04-30T06:26:28.420ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44732025-04-30T06:26:28.420ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44742025-04-30T06:26:28.421ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44752025-04-30T06:26:28.423ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44762025-04-30T06:26:28.428ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44772025-04-30T06:26:28.428ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44782025-04-30T06:26:28.428ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44792025-04-30T06:26:28.429ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44802025-04-30T06:26:28.483ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
44812025-04-30T06:26:28.483ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
44822025-04-30T06:26:28.483ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44832025-04-30T06:26:28.483ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44842025-04-30T06:26:28.483ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44852025-04-30T06:26:28.483ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
44862025-04-30T06:26:28.484ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44872025-04-30T06:26:28.484ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:48983 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
44882025-04-30T06:26:28.488ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:48983 req_id = b7fe5768-d2a5-4adf-a441-0d7879ea5456 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44892025-04-30T06:26:28.488ZINFOtest_disk_expunge (dropshot): request completed latency_us = 544 local_addr = [::1]:44559 method = GET remote_addr = [::1]:48983 req_id = b7fe5768-d2a5-4adf-a441-0d7879ea5456 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44902025-04-30T06:26:28.489ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44912025-04-30T06:26:28.495ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44922025-04-30T06:26:28.495ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44932025-04-30T06:26:28.495ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
44942025-04-30T06:26:28.496ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44952025-04-30T06:26:28.496ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7fe5768-d2a5-4adf-a441-0d7879ea5456", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
44962025-04-30T06:26:28.498ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44972025-04-30T06:26:28.498ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44982025-04-30T06:26:28.498ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
44992025-04-30T06:26:28.500ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
45002025-04-30T06:26:28.500ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45012025-04-30T06:26:28.500ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45022025-04-30T06:26:28.500ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45032025-04-30T06:26:28.501ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45042025-04-30T06:26:28.505ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45052025-04-30T06:26:28.505ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
45062025-04-30T06:26:28.506ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45072025-04-30T06:26:28.507ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45082025-04-30T06:26:28.512ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
45092025-04-30T06:26:28.512ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45102025-04-30T06:26:28.512ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45112025-04-30T06:26:28.513ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45122025-04-30T06:26:28.549ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
45132025-04-30T06:26:28.549ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
45142025-04-30T06:26:28.549ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 396.533677ms iteration = 9 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
45152025-04-30T06:26:28.550ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 10 reason = Signaled
45162025-04-30T06:26:28.550ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
45172025-04-30T06:26:28.550ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45182025-04-30T06:26:28.550ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45192025-04-30T06:26:28.550ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45202025-04-30T06:26:28.551ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45212025-04-30T06:26:28.554ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
45222025-04-30T06:26:28.554ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45232025-04-30T06:26:28.571ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
45242025-04-30T06:26:28.571ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45252025-04-30T06:26:28.571ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45262025-04-30T06:26:28.573ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45272025-04-30T06:26:28.602ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
45282025-04-30T06:26:28.602ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
45292025-04-30T06:26:28.603ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 41 7a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Az..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61790 req_id = d96ffabe-00dd-430b-9300-e32ed8b6c020
45302025-04-30T06:26:28.603ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16762,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61790 req_id = d96ffabe-00dd-430b-9300-e32ed8b6c020
45312025-04-30T06:26:28.604ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
45322025-04-30T06:26:28.604ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45332025-04-30T06:26:28.604ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61790 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d96ffabe-00dd-430b-9300-e32ed8b6c020
45342025-04-30T06:26:28.605ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45352025-04-30T06:26:28.605ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45362025-04-30T06:26:28.605ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45372025-04-30T06:26:28.605ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
45382025-04-30T06:26:28.606ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45392025-04-30T06:26:28.606ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3253, tv_nsec: 405593758 } })
45402025-04-30T06:26:28.606ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
45412025-04-30T06:26:28.607ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8f fe 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52625 req_id = 39521cbe-beba-4c40-9d31-16147e13eb34
45422025-04-30T06:26:28.607ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36862,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52625 req_id = 39521cbe-beba-4c40-9d31-16147e13eb34
45432025-04-30T06:26:28.607ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45442025-04-30T06:26:28.607ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52625 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 39521cbe-beba-4c40-9d31-16147e13eb34
45452025-04-30T06:26:28.687ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
45462025-04-30T06:26:28.687ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45472025-04-30T06:26:28.706ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45482025-04-30T06:26:28.706ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45492025-04-30T06:26:28.706ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
45502025-04-30T06:26:28.706ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45512025-04-30T06:26:28.707ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 95 6f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .o..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52322 req_id = 543ca0df-99e9-4273-81d9-e4e79525ce28
45522025-04-30T06:26:28.707ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38255,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52322 req_id = 543ca0df-99e9-4273-81d9-e4e79525ce28
45532025-04-30T06:26:28.707ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
45542025-04-30T06:26:28.707ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38255, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52322 req_id = 543ca0df-99e9-4273-81d9-e4e79525ce28
45552025-04-30T06:26:28.709ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
45562025-04-30T06:26:28.709ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
45572025-04-30T06:26:28.709ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
45582025-04-30T06:26:28.709ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45592025-04-30T06:26:28.709ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
45602025-04-30T06:26:28.710ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45612025-04-30T06:26:28.715ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "923e9781-a1a0-476f-a939-bfd0263d8a55", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45622025-04-30T06:26:28.715ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
45632025-04-30T06:26:28.719ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25f2f618-c6c7-43d7-80f7-96164df5b9bc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45642025-04-30T06:26:28.720ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
45652025-04-30T06:26:28.722ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01b1a7ad-747e-474c-a36f-0ed52483aec1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45662025-04-30T06:26:28.722ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
45672025-04-30T06:26:28.724ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c098f382-446d-4f1d-81a4-118055f9ec27", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45682025-04-30T06:26:28.724ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
45692025-04-30T06:26:28.727ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09315d8a-0b27-4239-aed7-d7dfda33e3ae", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45702025-04-30T06:26:28.728ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
45712025-04-30T06:26:28.730ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3d13c14-fe2c-4c7e-af96-003c97a1d5e8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45722025-04-30T06:26:28.731ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
45732025-04-30T06:26:28.735ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e90bc8e5-088f-4b5b-84e2-3e84489b3fc5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45742025-04-30T06:26:28.735ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
45752025-04-30T06:26:28.742ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
45762025-04-30T06:26:28.743ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85ee0b5d-0a09-43db-af46-92aeea480837", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45772025-04-30T06:26:28.744ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
45782025-04-30T06:26:28.745ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eedf121a-fdcc-4bb8-a0be-59443aee6afc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45792025-04-30T06:26:28.745ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
45802025-04-30T06:26:28.746ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6ada8d3-0d95-47a2-b6c1-c1a96d06e586", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45812025-04-30T06:26:28.746ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
45822025-04-30T06:26:28.747ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85ccadfd-1c57-45f7-821a-fd71d11f9567", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45832025-04-30T06:26:28.747ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
45842025-04-30T06:26:28.748ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86dc2451-917c-4104-8a46-a7f7814c6774", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45852025-04-30T06:26:28.749ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
45862025-04-30T06:26:28.750ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00863214-adb3-4871-aef2-9fd1a967b9c4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45872025-04-30T06:26:28.750ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
45882025-04-30T06:26:28.752ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "842245a8-759d-4947-a8c1-79983c3cb6c3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45892025-04-30T06:26:28.752ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
45902025-04-30T06:26:28.760ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1e0c124-ddd3-419b-a0c4-afadb1c28e64", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45912025-04-30T06:26:28.761ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
45922025-04-30T06:26:28.762ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15dd84b3-b554-4dcb-94b5-2963f361c550", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45932025-04-30T06:26:28.765ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
45942025-04-30T06:26:28.767ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "738dec39-2172-4041-bc2a-772a8a558998", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45952025-04-30T06:26:28.767ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
45962025-04-30T06:26:28.769ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c45ac759-9a7c-4950-a3bb-7837503343bc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45972025-04-30T06:26:28.769ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
45982025-04-30T06:26:28.771ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7fec67f-39b9-4dd5-b8e6-1116982477a5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
45992025-04-30T06:26:28.772ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
46002025-04-30T06:26:28.773ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ffecbc1-fc32-43e9-8705-b63dd1112fde", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46012025-04-30T06:26:28.773ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
46022025-04-30T06:26:28.774ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f8a2aba-e3ec-4028-8bd1-dcf09848ad97", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46032025-04-30T06:26:28.774ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
46042025-04-30T06:26:28.775ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f61684e-ef61-449b-ae96-dfd6fa4e9e90", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46052025-04-30T06:26:28.776ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
46062025-04-30T06:26:28.777ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62baf5c1-98bc-45e3-a98d-bcee830e1c9e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46072025-04-30T06:26:28.777ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
46082025-04-30T06:26:28.779ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "991ff0e1-7739-4759-8e72-04ce18c17a31", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46092025-04-30T06:26:28.781ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
46102025-04-30T06:26:28.783ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90b50ef3-6ff5-4ab4-8711-d14e4243e7b3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46112025-04-30T06:26:28.783ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
46122025-04-30T06:26:28.786ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbd1c453-3b93-4819-9aad-88b73100b2eb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46132025-04-30T06:26:28.786ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
46142025-04-30T06:26:28.788ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65b221fa-6c86-46e8-bb26-19e714149df0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46152025-04-30T06:26:28.788ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
46162025-04-30T06:26:28.793ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46172025-04-30T06:26:28.793ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46182025-04-30T06:26:28.793ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46192025-04-30T06:26:28.793ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
46202025-04-30T06:26:28.794ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46212025-04-30T06:26:28.794ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd0eb93c-68dd-46cf-bdb4-38cdfc432088", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46222025-04-30T06:26:28.794ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
46232025-04-30T06:26:28.797ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46242025-04-30T06:26:28.807ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46252025-04-30T06:26:28.807ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46262025-04-30T06:26:28.807ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
46272025-04-30T06:26:28.807ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46282025-04-30T06:26:28.813ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46292025-04-30T06:26:28.814ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
46302025-04-30T06:26:28.815ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46312025-04-30T06:26:28.815ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab412fac-3a56-41e5-b89b-05cb0be59bdb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46322025-04-30T06:26:28.816ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
46332025-04-30T06:26:28.820ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25fc9b0e-3cac-4b75-b48e-cd2a7bc3a84e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46342025-04-30T06:26:28.820ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
46352025-04-30T06:26:28.825ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a483d5a8-68f8-4e47-aafa-783fc0dd7973", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46362025-04-30T06:26:28.825ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
46372025-04-30T06:26:28.829ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c714aef3-cd8c-4ef4-b537-cd73717c29ca", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46382025-04-30T06:26:28.829ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
46392025-04-30T06:26:28.830ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e6d74d8-7e67-4797-aa9e-8a8d79ff979b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46402025-04-30T06:26:28.831ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
46412025-04-30T06:26:28.832ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f43468c3-b064-41c1-9e41-d4b29c383938", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46422025-04-30T06:26:28.832ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
46432025-04-30T06:26:28.834ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f663c2c-2e2a-4586-8e8d-cfb406e686e2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46442025-04-30T06:26:28.834ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
46452025-04-30T06:26:28.837ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71d031e9-5279-4974-9855-2023877f5bd2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46462025-04-30T06:26:28.837ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
46472025-04-30T06:26:28.843ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "67512990-35a8-4aca-9db5-9fe731a63f21", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46482025-04-30T06:26:28.843ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
46492025-04-30T06:26:28.846ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d5d392c-e447-45d6-bd8c-17e7d58774e3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46502025-04-30T06:26:28.847ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
46512025-04-30T06:26:28.850ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
46522025-04-30T06:26:28.852ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "492c6bd3-cf70-4ffe-8c13-5c3a2c1cfb49", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46532025-04-30T06:26:28.853ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
46542025-04-30T06:26:28.855ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1081533-2ec5-45f6-8c8d-c1a20f8b44f8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46552025-04-30T06:26:28.855ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
46562025-04-30T06:26:28.857ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2520194-c4d6-4f48-aa2d-55caab65e947", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46572025-04-30T06:26:28.857ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
46582025-04-30T06:26:28.859ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3151b3e3-8026-4773-bbbb-d782ef236193", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46592025-04-30T06:26:28.859ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
46602025-04-30T06:26:28.860ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7379d9dc-ae8a-4595-8624-1d6eb39ee40b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46612025-04-30T06:26:28.860ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
46622025-04-30T06:26:28.861ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e403e4c7-de03-4f03-aa49-b42df3f797e1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46632025-04-30T06:26:28.862ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
46642025-04-30T06:26:28.863ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d91d9e48-b5f2-4d38-93b6-054527758a77", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46652025-04-30T06:26:28.863ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46662025-04-30T06:26:28.863ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46672025-04-30T06:26:28.863ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46682025-04-30T06:26:28.863ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46692025-04-30T06:26:28.864ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46702025-04-30T06:26:28.871ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46712025-04-30T06:26:28.876ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46722025-04-30T06:26:28.876ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46732025-04-30T06:26:28.876ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46742025-04-30T06:26:28.876ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46752025-04-30T06:26:28.876ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46762025-04-30T06:26:28.877ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46772025-04-30T06:26:28.878ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
46782025-04-30T06:26:28.878ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46792025-04-30T06:26:28.879ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = b0a5a4d2-f272-4cdb-9313-a25e78c521cc uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46802025-04-30T06:26:28.879ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = b0a5a4d2-f272-4cdb-9313-a25e78c521cc resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46812025-04-30T06:26:28.879ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = b0a5a4d2-f272-4cdb-9313-a25e78c521cc roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46822025-04-30T06:26:28.879ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = b0a5a4d2-f272-4cdb-9313-a25e78c521cc resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46832025-04-30T06:26:28.880ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46842025-04-30T06:26:28.883ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46852025-04-30T06:26:28.883ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46862025-04-30T06:26:28.883ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46872025-04-30T06:26:28.886ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46882025-04-30T06:26:28.888ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 9612 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = b0a5a4d2-f272-4cdb-9313-a25e78c521cc response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46892025-04-30T06:26:28.889ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0a5a4d2-f272-4cdb-9313-a25e78c521cc", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
46902025-04-30T06:26:28.889ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46912025-04-30T06:26:28.889ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 7b171b3e-56c3-4f5b-ba14-238658b435fd uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46922025-04-30T06:26:28.890ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 7b171b3e-56c3-4f5b-ba14-238658b435fd resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46932025-04-30T06:26:28.890ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 7b171b3e-56c3-4f5b-ba14-238658b435fd roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46942025-04-30T06:26:28.890ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 7b171b3e-56c3-4f5b-ba14-238658b435fd resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46952025-04-30T06:26:29.008ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
46962025-04-30T06:26:29.008ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
46972025-04-30T06:26:29.009ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 120040 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 7b171b3e-56c3-4f5b-ba14-238658b435fd response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
46982025-04-30T06:26:29.011ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46992025-04-30T06:26:29.012ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47002025-04-30T06:26:29.012ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47012025-04-30T06:26:29.013ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
47022025-04-30T06:26:29.015ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47032025-04-30T06:26:29.016ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:41567 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
47042025-04-30T06:26:29.016ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47052025-04-30T06:26:29.017ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47062025-04-30T06:26:29.018ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b171b3e-56c3-4f5b-ba14-238658b435fd", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
47072025-04-30T06:26:29.018ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
47082025-04-30T06:26:29.018ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47092025-04-30T06:26:29.018ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
47102025-04-30T06:26:29.019ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
47112025-04-30T06:26:29.020ZDEBGtest_disk_expunge (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
47122025-04-30T06:26:29.020ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = SELECT * FROM oximeter.timeseries_schema FORMAT Native;
47132025-04-30T06:26:29.021ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:41567 req_id = 9fabae8e-eec2-48ec-8395-66f792a69e9f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
47142025-04-30T06:26:29.022ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1381 local_addr = [::1]:44559 method = GET remote_addr = [::1]:41567 req_id = 9fabae8e-eec2-48ec-8395-66f792a69e9f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
47152025-04-30T06:26:29.024ZTRACtest_disk_expunge (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
47162025-04-30T06:26:29.024ZDEBGtest_disk_expunge (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("integration_target:integration_metric"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "metric_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "target_name", field_type: String, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:26:29.020727216Z } timeseries_name = integration_target:integration_metric
47172025-04-30T06:26:29.025ZDEBGtest_disk_expunge (clickhouse-client): inserting 1 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
47182025-04-30T06:26:29.025ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 6 n_rows = 1 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
47192025-04-30T06:26:29.025ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fabae8e-eec2-48ec-8395-66f792a69e9f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:28 GMT"} })
47202025-04-30T06:26:29.028ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
47212025-04-30T06:26:29.050ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
47222025-04-30T06:26:29.050ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
47232025-04-30T06:26:29.053ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
47242025-04-30T06:26:29.053ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47252025-04-30T06:26:29.053ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47262025-04-30T06:26:29.053ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47272025-04-30T06:26:29.053ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47282025-04-30T06:26:29.055ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47292025-04-30T06:26:29.061ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47302025-04-30T06:26:29.062ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47312025-04-30T06:26:29.062ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
47322025-04-30T06:26:29.063ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47332025-04-30T06:26:29.069ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47342025-04-30T06:26:29.069ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47352025-04-30T06:26:29.071ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
47362025-04-30T06:26:29.079ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47372025-04-30T06:26:29.079ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47382025-04-30T06:26:29.079ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47392025-04-30T06:26:29.079ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47402025-04-30T06:26:29.080ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47412025-04-30T06:26:29.088ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47422025-04-30T06:26:29.089ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47432025-04-30T06:26:29.089ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47442025-04-30T06:26:29.089ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47452025-04-30T06:26:29.090ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47462025-04-30T06:26:29.090ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
47472025-04-30T06:26:29.093ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47482025-04-30T06:26:29.094ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
47492025-04-30T06:26:29.094ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
47502025-04-30T06:26:29.106ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
47512025-04-30T06:26:29.107ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47522025-04-30T06:26:29.120ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
47532025-04-30T06:26:29.165ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
47542025-04-30T06:26:29.165ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
47552025-04-30T06:26:29.165ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 615.857014ms iteration = 10 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
47562025-04-30T06:26:29.166ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 11 reason = Signaled
47572025-04-30T06:26:29.166ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
47582025-04-30T06:26:29.166ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47592025-04-30T06:26:29.166ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47602025-04-30T06:26:29.166ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47612025-04-30T06:26:29.167ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47622025-04-30T06:26:29.173ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47632025-04-30T06:26:29.181ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47642025-04-30T06:26:29.181ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47652025-04-30T06:26:29.181ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47662025-04-30T06:26:29.181ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47672025-04-30T06:26:29.188ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
47682025-04-30T06:26:29.201ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
47692025-04-30T06:26:29.204ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d8 9a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50703 req_id = c633ca19-c070-476b-b8a3-0c4b55264733
47702025-04-30T06:26:29.204ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55450,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50703 req_id = c633ca19-c070-476b-b8a3-0c4b55264733
47712025-04-30T06:26:29.204ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
47722025-04-30T06:26:29.204ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47732025-04-30T06:26:29.204ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50703 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c633ca19-c070-476b-b8a3-0c4b55264733
47742025-04-30T06:26:29.205ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
47752025-04-30T06:26:29.205ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3254, tv_nsec: 4554257 } })
47762025-04-30T06:26:29.207ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
47772025-04-30T06:26:29.207ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c7 34 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .4..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52963 req_id = 0fb76179-7db7-46aa-913d-396477048727
47782025-04-30T06:26:29.207ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50996,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52963 req_id = 0fb76179-7db7-46aa-913d-396477048727
47792025-04-30T06:26:29.207ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47802025-04-30T06:26:29.207ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52963 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0fb76179-7db7-46aa-913d-396477048727
47812025-04-30T06:26:29.275ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
47822025-04-30T06:26:29.275ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47832025-04-30T06:26:29.275ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47842025-04-30T06:26:29.275ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47852025-04-30T06:26:29.275ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
47862025-04-30T06:26:29.276ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47872025-04-30T06:26:29.277ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 67 d9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c g............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49601 req_id = 0ba4c890-3a40-430f-8e11-f0d73a122a8e
47882025-04-30T06:26:29.277ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26585,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49601 req_id = 0ba4c890-3a40-430f-8e11-f0d73a122a8e
47892025-04-30T06:26:29.277ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
47902025-04-30T06:26:29.277ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 26585, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49601 req_id = 0ba4c890-3a40-430f-8e11-f0d73a122a8e
47912025-04-30T06:26:29.278ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
47922025-04-30T06:26:29.278ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
47932025-04-30T06:26:29.278ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
47942025-04-30T06:26:29.279ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47952025-04-30T06:26:29.288ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47962025-04-30T06:26:29.288ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47972025-04-30T06:26:29.288ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
47982025-04-30T06:26:29.289ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47992025-04-30T06:26:29.290ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "bad6619c-82be-40b1-b62c-1f12cb301d8e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48002025-04-30T06:26:29.291ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
48012025-04-30T06:26:29.292ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ef99269-110d-474a-9155-cce90f5deb40", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48022025-04-30T06:26:29.293ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
48032025-04-30T06:26:29.294ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48042025-04-30T06:26:29.294ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
48052025-04-30T06:26:29.294ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48062025-04-30T06:26:29.297ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cff9ff9-bbf7-4a00-8904-2b4bcadfe224", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48072025-04-30T06:26:29.297ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
48082025-04-30T06:26:29.300ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1fde7ea-2c7c-4ccb-a021-757adb923e77", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48092025-04-30T06:26:29.300ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
48102025-04-30T06:26:29.304ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "918ab6a8-a84d-4362-b89c-cb15fc0182b8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48112025-04-30T06:26:29.304ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
48122025-04-30T06:26:29.308ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eea85f51-ee28-46aa-8851-2d675677a64a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48132025-04-30T06:26:29.308ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
48142025-04-30T06:26:29.315ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1af02ae-0111-4b17-9519-5aab4de5be22", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48152025-04-30T06:26:29.315ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
48162025-04-30T06:26:29.319ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2970de03-ae08-4cff-a253-2b75b68cd86d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48172025-04-30T06:26:29.319ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
48182025-04-30T06:26:29.320ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2e4955d-f6bc-4d04-8089-d9fc98a7d0b6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48192025-04-30T06:26:29.321ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
48202025-04-30T06:26:29.322ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e798a2d5-8982-417a-a84b-ad515b7d2b26", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48212025-04-30T06:26:29.322ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
48222025-04-30T06:26:29.324ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad5f923a-5e74-4aed-96da-f158df045851", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48232025-04-30T06:26:29.324ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
48242025-04-30T06:26:29.326ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "11924ace-2053-423c-9e11-18a17e88628f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48252025-04-30T06:26:29.326ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
48262025-04-30T06:26:29.328ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68ff29ac-d9f4-4b7f-aab7-401455a01dd3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48272025-04-30T06:26:29.328ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
48282025-04-30T06:26:29.334ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
48292025-04-30T06:26:29.347ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88907efe-e30c-43ef-8941-e94574195b29", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48302025-04-30T06:26:29.347ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
48312025-04-30T06:26:29.349ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3a0fb45-4f52-40b6-aff9-35e52ef4a135", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48322025-04-30T06:26:29.349ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
48332025-04-30T06:26:29.351ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4832ddb2-335a-4ddd-8f3f-ff3c9f110009", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48342025-04-30T06:26:29.352ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
48352025-04-30T06:26:29.354ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c779faf-4bdf-4d5c-ad6e-af3acd51bda8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48362025-04-30T06:26:29.354ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
48372025-04-30T06:26:29.358ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4156a7a2-9293-432f-a5e6-08ed895ec1ea", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48382025-04-30T06:26:29.358ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
48392025-04-30T06:26:29.360ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71948107-4e01-43a1-af50-5ad70322da9f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48402025-04-30T06:26:29.360ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
48412025-04-30T06:26:29.361ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05c18aae-829f-4888-a489-c21d505dede2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48422025-04-30T06:26:29.361ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
48432025-04-30T06:26:29.364ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52bfc6be-0d77-4121-91ec-a2f7a81e164d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48442025-04-30T06:26:29.364ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
48452025-04-30T06:26:29.372ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6aa7c935-ec61-4067-a0d7-fe2377495c9a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48462025-04-30T06:26:29.372ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
48472025-04-30T06:26:29.375ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e969c01f-e2e8-4882-a1d4-8d7505d78c8b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48482025-04-30T06:26:29.375ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
48492025-04-30T06:26:29.377ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fcd4f9e-72b1-4ca3-95e3-8f0eda1225f9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48502025-04-30T06:26:29.377ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
48512025-04-30T06:26:29.381ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20a7d968-9aa9-49f6-ba9f-484e751b6a8d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48522025-04-30T06:26:29.381ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
48532025-04-30T06:26:29.384ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13bd651e-1cbe-44fa-b8dd-d851aa5fa32e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48542025-04-30T06:26:29.385ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
48552025-04-30T06:26:29.385ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48562025-04-30T06:26:29.385ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48572025-04-30T06:26:29.385ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48582025-04-30T06:26:29.385ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
48592025-04-30T06:26:29.387ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48602025-04-30T06:26:29.396ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48612025-04-30T06:26:29.403ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48622025-04-30T06:26:29.403ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48632025-04-30T06:26:29.403ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
48642025-04-30T06:26:29.403ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48652025-04-30T06:26:29.404ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d3794a1-87e5-441d-8847-036221121edf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48662025-04-30T06:26:29.405ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
48672025-04-30T06:26:29.407ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48682025-04-30T06:26:29.407ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
48692025-04-30T06:26:29.407ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48702025-04-30T06:26:29.408ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e14685d-d41f-4dee-b224-af7ec405ac64", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48712025-04-30T06:26:29.408ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
48722025-04-30T06:26:29.423ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe489da6-bc7a-48d4-a224-b65acb88211e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48732025-04-30T06:26:29.426ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
48742025-04-30T06:26:29.430ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afbf7f42-c114-48e7-932e-b27f9a333214", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48752025-04-30T06:26:29.430ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
48762025-04-30T06:26:29.433ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83846e02-44b1-49e1-9dfe-78e3da13a2ae", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48772025-04-30T06:26:29.433ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
48782025-04-30T06:26:29.435ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2bc07fb-6d6b-4576-bcf4-ccea30857976", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48792025-04-30T06:26:29.435ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
48802025-04-30T06:26:29.438ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c4677dc-23c9-4df8-bc38-b755a87a7f4d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48812025-04-30T06:26:29.438ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
48822025-04-30T06:26:29.441ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "add9d654-3a50-442d-a2b9-96ecce5bec29", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48832025-04-30T06:26:29.441ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
48842025-04-30T06:26:29.444ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16f4cd58-5d21-4ebf-9b08-ddd7547b0118", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48852025-04-30T06:26:29.444ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
48862025-04-30T06:26:29.451ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "496cadbc-f833-4cd6-885c-f3f5dc8a2756", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48872025-04-30T06:26:29.452ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
48882025-04-30T06:26:29.460ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
48892025-04-30T06:26:29.463ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c33f71b-b09c-4d68-8ba5-4b17367988ac", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48902025-04-30T06:26:29.463ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
48912025-04-30T06:26:29.467ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9dcae8a-3d06-4747-b165-b84024d8bfb3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48922025-04-30T06:26:29.467ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
48932025-04-30T06:26:29.472ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21872433-0fab-47f6-9b04-65e3284b160c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48942025-04-30T06:26:29.473ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
48952025-04-30T06:26:29.477ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfd2b8c0-ba75-4c7a-9115-78d49eeb23ae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48962025-04-30T06:26:29.478ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
48972025-04-30T06:26:29.482ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1c7abb7-8420-4767-9eaf-3f433859a54e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
48982025-04-30T06:26:29.483ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
48992025-04-30T06:26:29.484ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93370f99-3972-4769-bbd3-38991e3c8191", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
49002025-04-30T06:26:29.485ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
49012025-04-30T06:26:29.487ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c8a8533-91a7-411f-a4be-972accdcf33a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
49022025-04-30T06:26:29.488ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
49032025-04-30T06:26:29.490ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4158c8be-45a3-4e71-a11e-ebba454fb70c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
49042025-04-30T06:26:29.490ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
49052025-04-30T06:26:29.491ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff8f27ee-4390-46d4-aaa9-93e75374b127", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
49062025-04-30T06:26:29.491ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
49072025-04-30T06:26:29.491ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49082025-04-30T06:26:29.491ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49092025-04-30T06:26:29.491ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49102025-04-30T06:26:29.492ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49112025-04-30T06:26:29.503ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49122025-04-30T06:26:29.509ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
49132025-04-30T06:26:29.509ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
49142025-04-30T06:26:29.509ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49152025-04-30T06:26:29.509ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49162025-04-30T06:26:29.509ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49172025-04-30T06:26:29.509ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49182025-04-30T06:26:29.512ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49192025-04-30T06:26:29.512ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49202025-04-30T06:26:29.512ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49212025-04-30T06:26:29.512ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
49222025-04-30T06:26:29.513ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49232025-04-30T06:26:29.515ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49242025-04-30T06:26:29.536ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
49252025-04-30T06:26:29.537ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49262025-04-30T06:26:29.537ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49272025-04-30T06:26:29.537ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49282025-04-30T06:26:29.540ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49292025-04-30T06:26:29.550ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49302025-04-30T06:26:29.550ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49312025-04-30T06:26:29.551ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
49322025-04-30T06:26:29.551ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49332025-04-30T06:26:29.554ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49342025-04-30T06:26:29.555ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
49352025-04-30T06:26:29.555ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49362025-04-30T06:26:29.663ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
49372025-04-30T06:26:29.663ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
49382025-04-30T06:26:29.666ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:62409 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
49392025-04-30T06:26:29.667ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:62409 req_id = 697b43e1-3f5b-4c48-8421-5af641efb3a9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
49402025-04-30T06:26:29.669ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1199 local_addr = [::1]:44559 method = GET remote_addr = [::1]:62409 req_id = 697b43e1-3f5b-4c48-8421-5af641efb3a9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
49412025-04-30T06:26:29.669ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "697b43e1-3f5b-4c48-8421-5af641efb3a9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
49422025-04-30T06:26:29.673ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
49432025-04-30T06:26:29.673ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
49442025-04-30T06:26:29.673ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
49452025-04-30T06:26:29.675ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49462025-04-30T06:26:29.676ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49472025-04-30T06:26:29.676ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49482025-04-30T06:26:29.676ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49492025-04-30T06:26:29.677ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49502025-04-30T06:26:29.681ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49512025-04-30T06:26:29.686ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49522025-04-30T06:26:29.687ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49532025-04-30T06:26:29.687ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49542025-04-30T06:26:29.688ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49552025-04-30T06:26:29.764ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
49562025-04-30T06:26:29.819ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49572025-04-30T06:26:29.819ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49582025-04-30T06:26:29.819ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49592025-04-30T06:26:29.819ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
49602025-04-30T06:26:29.820ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49612025-04-30T06:26:29.824ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49622025-04-30T06:26:29.829ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49632025-04-30T06:26:29.829ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49642025-04-30T06:26:29.830ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
49652025-04-30T06:26:29.830ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49662025-04-30T06:26:29.833ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
49672025-04-30T06:26:29.833ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
49682025-04-30T06:26:29.833ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 667.706942ms iteration = 11 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
49692025-04-30T06:26:29.833ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 12 reason = Signaled
49702025-04-30T06:26:29.833ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49712025-04-30T06:26:29.833ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49722025-04-30T06:26:29.834ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49732025-04-30T06:26:29.834ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49742025-04-30T06:26:29.834ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49752025-04-30T06:26:29.836ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49762025-04-30T06:26:29.836ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
49772025-04-30T06:26:29.836ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49782025-04-30T06:26:29.836ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49792025-04-30T06:26:29.840ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49802025-04-30T06:26:29.841ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49812025-04-30T06:26:29.841ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49822025-04-30T06:26:29.841ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49832025-04-30T06:26:29.844ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
49842025-04-30T06:26:29.844ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
49852025-04-30T06:26:29.845ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cd d3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55900 req_id = 3d7c53a9-ea10-451f-90df-492cc6076a7f
49862025-04-30T06:26:29.846ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52691,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55900 req_id = 3d7c53a9-ea10-451f-90df-492cc6076a7f
49872025-04-30T06:26:29.846ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
49882025-04-30T06:26:29.846ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49892025-04-30T06:26:29.847ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55900 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3d7c53a9-ea10-451f-90df-492cc6076a7f
49902025-04-30T06:26:29.848ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3254, tv_nsec: 647926174 } })
49912025-04-30T06:26:29.849ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
49922025-04-30T06:26:29.851ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 96 73 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .s..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51464 req_id = 7d1539de-d7fb-4c41-8d6e-58c52c343612
49932025-04-30T06:26:29.852ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38515,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51464 req_id = 7d1539de-d7fb-4c41-8d6e-58c52c343612
49942025-04-30T06:26:29.852ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49952025-04-30T06:26:29.852ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51464 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7d1539de-d7fb-4c41-8d6e-58c52c343612
49962025-04-30T06:26:29.919ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
49972025-04-30T06:26:29.921ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 70 1f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c p............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51213 req_id = 89ef4ca2-ec69-42bf-b6b1-5c53aa77e378
49982025-04-30T06:26:29.921ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28703,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51213 req_id = 89ef4ca2-ec69-42bf-b6b1-5c53aa77e378
49992025-04-30T06:26:29.921ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
50002025-04-30T06:26:29.921ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28703, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51213 req_id = 89ef4ca2-ec69-42bf-b6b1-5c53aa77e378
50012025-04-30T06:26:29.922ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
50022025-04-30T06:26:29.922ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
50032025-04-30T06:26:29.922ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
50042025-04-30T06:26:29.925ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "259bc856-2cc1-4ad7-a9a4-846dbda8c8fb", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50052025-04-30T06:26:29.925ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
50062025-04-30T06:26:29.927ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb0d414b-7535-49f7-a0fa-d0c13e2237fc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50072025-04-30T06:26:29.927ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
50082025-04-30T06:26:29.930ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d61e1eaa-3895-4f83-af8b-7deb5b144e39", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50092025-04-30T06:26:29.930ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
50102025-04-30T06:26:29.932ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e60b5239-1f73-425b-b08d-ca3d9cea4658", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50112025-04-30T06:26:29.932ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
50122025-04-30T06:26:29.936ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7765e5f4-6af4-4337-9093-bae18fca88b4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50132025-04-30T06:26:29.936ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
50142025-04-30T06:26:29.945ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
50152025-04-30T06:26:29.951ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "366d4b81-f332-4d01-bd7b-390b76a44b24", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50162025-04-30T06:26:29.951ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
50172025-04-30T06:26:29.953ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3cfa87e-7b94-41bc-b3a1-6cc6a5be203a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50182025-04-30T06:26:29.953ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
50192025-04-30T06:26:29.956ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50202025-04-30T06:26:29.956ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50212025-04-30T06:26:29.956ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ebfe138-2dd1-4f0a-b827-3c292c1982ed", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50222025-04-30T06:26:29.956ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
50232025-04-30T06:26:29.956ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
50242025-04-30T06:26:29.957ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
50252025-04-30T06:26:29.958ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07a21d8b-bc0b-4d51-a37d-7103f2db8d45", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50262025-04-30T06:26:29.958ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
50272025-04-30T06:26:29.960ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae01997c-67c5-4858-9019-e14653ddf450", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50282025-04-30T06:26:29.960ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
50292025-04-30T06:26:29.961ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
50302025-04-30T06:26:29.962ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
50312025-04-30T06:26:29.964ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21b1a6ab-7c66-45da-a3b9-8810ee29cb82", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50322025-04-30T06:26:29.965ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
50332025-04-30T06:26:29.967ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
50342025-04-30T06:26:29.967ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50352025-04-30T06:26:29.968ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c312968-6731-4c14-8d38-ccc7b304c3be", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50362025-04-30T06:26:29.968ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
50372025-04-30T06:26:29.970ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4c7aaf8-4813-478e-91e6-32972327ce34", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50382025-04-30T06:26:29.970ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
50392025-04-30T06:26:29.972ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4c53c2d-bc75-40ee-8af3-3257e1b1b06c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50402025-04-30T06:26:29.972ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
50412025-04-30T06:26:29.976ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2e37675-051b-46c4-b754-05a262446be8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50422025-04-30T06:26:29.976ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
50432025-04-30T06:26:29.978ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dfe97ba-4e16-4e59-8396-d57fca1506f3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50442025-04-30T06:26:29.978ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
50452025-04-30T06:26:29.980ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "facfee5c-e912-4daa-b478-ec4e220d885e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50462025-04-30T06:26:29.980ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
50472025-04-30T06:26:29.981ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18425818-0f26-40a2-b7e0-07c867d98800", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50482025-04-30T06:26:29.982ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
50492025-04-30T06:26:29.984ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7c1a270-46e7-4bb6-936f-a2c8e5859e3a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50502025-04-30T06:26:29.984ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
50512025-04-30T06:26:29.985ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d31e877-2505-4e63-9ac5-9530a428834a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50522025-04-30T06:26:29.985ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
50532025-04-30T06:26:29.986ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15a7d799-6fa3-4257-98b4-006fee661c57", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50542025-04-30T06:26:29.986ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
50552025-04-30T06:26:29.987ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1be8c934-5cb3-458c-9ee6-fd42e41c6023", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50562025-04-30T06:26:29.987ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
50572025-04-30T06:26:29.989ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55ac24d1-0928-4d47-afb0-440311154b89", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50582025-04-30T06:26:29.989ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
50592025-04-30T06:26:29.991ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7aec7a9-65d7-4e19-9caf-d0a662af0065", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50602025-04-30T06:26:29.991ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
50612025-04-30T06:26:29.995ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "40f25fd4-d2ef-47e2-b323-3bb11a7e5d30", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50622025-04-30T06:26:29.995ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
50632025-04-30T06:26:29.996ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50642025-04-30T06:26:29.997ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50652025-04-30T06:26:29.997ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50662025-04-30T06:26:29.997ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
50672025-04-30T06:26:29.998ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50682025-04-30T06:26:30.003ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50692025-04-30T06:26:30.017ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50702025-04-30T06:26:30.018ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50712025-04-30T06:26:30.018ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
50722025-04-30T06:26:30.018ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50732025-04-30T06:26:30.021ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd66c89d-8fb8-4888-9b71-b677864728de", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:29 GMT"} })
50742025-04-30T06:26:30.021ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
50752025-04-30T06:26:30.023ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50762025-04-30T06:26:30.023ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
50772025-04-30T06:26:30.026ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50782025-04-30T06:26:30.043ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50792025-04-30T06:26:30.047ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50802025-04-30T06:26:30.051ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50812025-04-30T06:26:30.052ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50822025-04-30T06:26:30.052ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50832025-04-30T06:26:30.052ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50842025-04-30T06:26:30.061ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
50852025-04-30T06:26:30.083ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e621c04-3b72-48d0-a1ad-94df5991cc7c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
50862025-04-30T06:26:30.083ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
50872025-04-30T06:26:30.091ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
50882025-04-30T06:26:30.098ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25bf941f-b4c3-45b8-b0fc-b0187e1c1378", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
50892025-04-30T06:26:30.098ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
50902025-04-30T06:26:30.102ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fcc0d93-0a14-4644-9508-82876757ff30", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
50912025-04-30T06:26:30.102ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
50922025-04-30T06:26:30.105ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02e6a7e2-0401-4c91-a71a-dd8820526eca", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
50932025-04-30T06:26:30.105ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
50942025-04-30T06:26:30.106ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92f14309-e173-4506-bd60-cccdb1918111", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
50952025-04-30T06:26:30.107ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
50962025-04-30T06:26:30.109ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81a8f68d-1a50-4022-80d2-cc50774fcc50", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
50972025-04-30T06:26:30.109ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
50982025-04-30T06:26:30.110ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33977245-82e9-4fc0-abd8-7fa4cacb2c4c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
50992025-04-30T06:26:30.110ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
51002025-04-30T06:26:30.113ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6831801-482f-4bc0-9d2b-648a63dc0887", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51012025-04-30T06:26:30.113ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
51022025-04-30T06:26:30.114ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "26eaefad-4787-4ae6-bdd9-0c56e4af23da", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51032025-04-30T06:26:30.115ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
51042025-04-30T06:26:30.131ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3c26b90-73e3-4652-be27-495e15f0a613", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51052025-04-30T06:26:30.132ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
51062025-04-30T06:26:30.137ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0567bc85-ed1e-4232-9af2-9757daff270b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51072025-04-30T06:26:30.138ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
51082025-04-30T06:26:30.144ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7c7961b-067f-4875-9b93-8df9586b970e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51092025-04-30T06:26:30.145ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
51102025-04-30T06:26:30.149ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51112025-04-30T06:26:30.149ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51122025-04-30T06:26:30.149ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51132025-04-30T06:26:30.149ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
51142025-04-30T06:26:30.151ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51152025-04-30T06:26:30.153ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9475ec0f-f3ce-4d74-850f-36ebdf0c7ae7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51162025-04-30T06:26:30.153ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
51172025-04-30T06:26:30.155ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f712839-4b9a-4698-9d0c-76b6289a56a1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51182025-04-30T06:26:30.155ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
51192025-04-30T06:26:30.156ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51202025-04-30T06:26:30.164ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51212025-04-30T06:26:30.164ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51222025-04-30T06:26:30.164ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
51232025-04-30T06:26:30.165ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51242025-04-30T06:26:30.173ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51252025-04-30T06:26:30.174ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
51262025-04-30T06:26:30.174ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51272025-04-30T06:26:30.176ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1ab4ccf-1b07-44b9-b386-f00f97f8d84e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51282025-04-30T06:26:30.176ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
51292025-04-30T06:26:30.178ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff3276a2-706a-469b-b809-bd01c89441da", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51302025-04-30T06:26:30.178ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
51312025-04-30T06:26:30.179ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6e98ab7-f382-4d31-88f2-85dbc8f7c7b6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51322025-04-30T06:26:30.179ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
51332025-04-30T06:26:30.181ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fee3f1d-838d-49b1-bf70-497e3ef8fd96", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51342025-04-30T06:26:30.181ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
51352025-04-30T06:26:30.182ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b1d3178-58b0-42b8-a6ef-ee3589059205", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51362025-04-30T06:26:30.184ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
51372025-04-30T06:26:30.184ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51382025-04-30T06:26:30.184ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51392025-04-30T06:26:30.184ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51402025-04-30T06:26:30.186ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51412025-04-30T06:26:30.190ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51422025-04-30T06:26:30.197ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
51432025-04-30T06:26:30.197ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
51442025-04-30T06:26:30.197ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51452025-04-30T06:26:30.197ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51462025-04-30T06:26:30.197ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51472025-04-30T06:26:30.198ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51482025-04-30T06:26:30.202ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51492025-04-30T06:26:30.206ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
51502025-04-30T06:26:30.206ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51512025-04-30T06:26:30.206ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51522025-04-30T06:26:30.207ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51532025-04-30T06:26:30.256ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
51542025-04-30T06:26:30.256ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
51552025-04-30T06:26:30.257ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:56261 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
51562025-04-30T06:26:30.258ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:56261 req_id = d07b56b7-2aa0-427d-8eb2-b866ee2b7a34 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51572025-04-30T06:26:30.258ZINFOtest_disk_expunge (dropshot): request completed latency_us = 498 local_addr = [::1]:44559 method = GET remote_addr = [::1]:56261 req_id = d07b56b7-2aa0-427d-8eb2-b866ee2b7a34 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51582025-04-30T06:26:30.259ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d07b56b7-2aa0-427d-8eb2-b866ee2b7a34", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
51592025-04-30T06:26:30.262ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51602025-04-30T06:26:30.262ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51612025-04-30T06:26:30.262ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
51622025-04-30T06:26:30.264ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51632025-04-30T06:26:30.264ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51642025-04-30T06:26:30.265ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51652025-04-30T06:26:30.265ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51662025-04-30T06:26:30.265ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51672025-04-30T06:26:30.276ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
51682025-04-30T06:26:30.276ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51692025-04-30T06:26:30.283ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51702025-04-30T06:26:30.283ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51712025-04-30T06:26:30.283ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51722025-04-30T06:26:30.284ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51732025-04-30T06:26:30.316ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
51742025-04-30T06:26:30.317ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
51752025-04-30T06:26:30.317ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 483.17293ms iteration = 12 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
51762025-04-30T06:26:30.317ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 13 reason = Signaled
51772025-04-30T06:26:30.317ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51782025-04-30T06:26:30.317ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51792025-04-30T06:26:30.317ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51802025-04-30T06:26:30.317ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51812025-04-30T06:26:30.317ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51822025-04-30T06:26:30.320ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51832025-04-30T06:26:30.323ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51842025-04-30T06:26:30.323ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51852025-04-30T06:26:30.323ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51862025-04-30T06:26:30.324ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51872025-04-30T06:26:30.327ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51882025-04-30T06:26:30.327ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51892025-04-30T06:26:30.327ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51902025-04-30T06:26:30.327ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
51912025-04-30T06:26:30.328ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51922025-04-30T06:26:30.328ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
51932025-04-30T06:26:30.328ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
51942025-04-30T06:26:30.329ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 48 f5 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 H............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53774 req_id = c42608af-fed1-4509-9ad1-082f623ea3ea
51952025-04-30T06:26:30.329ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18677,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53774 req_id = c42608af-fed1-4509-9ad1-082f623ea3ea
51962025-04-30T06:26:30.329ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
51972025-04-30T06:26:30.329ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51982025-04-30T06:26:30.329ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53774 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c42608af-fed1-4509-9ad1-082f623ea3ea
51992025-04-30T06:26:30.330ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3255, tv_nsec: 129429458 } })
52002025-04-30T06:26:30.330ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
52012025-04-30T06:26:30.331ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0b 83 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54341 req_id = 55e5ce40-dc77-4450-8d6a-232d56b0ed05
52022025-04-30T06:26:30.331ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2947,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54341 req_id = 55e5ce40-dc77-4450-8d6a-232d56b0ed05
52032025-04-30T06:26:30.331ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
52042025-04-30T06:26:30.331ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54341 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 55e5ce40-dc77-4450-8d6a-232d56b0ed05
52052025-04-30T06:26:30.331ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52062025-04-30T06:26:30.337ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52072025-04-30T06:26:30.337ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52082025-04-30T06:26:30.337ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
52092025-04-30T06:26:30.337ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52102025-04-30T06:26:30.396ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
52112025-04-30T06:26:30.402ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 49 61 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c Ia..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57631 req_id = 4f402c93-fe34-450d-8322-2e8888d7f81b
52122025-04-30T06:26:30.402ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18785,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57631 req_id = 4f402c93-fe34-450d-8322-2e8888d7f81b
52132025-04-30T06:26:30.402ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
52142025-04-30T06:26:30.402ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18785, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57631 req_id = 4f402c93-fe34-450d-8322-2e8888d7f81b
52152025-04-30T06:26:30.403ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52162025-04-30T06:26:30.403ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
52172025-04-30T06:26:30.404ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52182025-04-30T06:26:30.404ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
52192025-04-30T06:26:30.405ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
52202025-04-30T06:26:30.405ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
52212025-04-30T06:26:30.409ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "101242d3-727d-4226-b126-a736ccc91188", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52222025-04-30T06:26:30.410ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
52232025-04-30T06:26:30.412ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9909624b-6d04-4ccf-ae12-6a83a732f082", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52242025-04-30T06:26:30.412ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
52252025-04-30T06:26:30.416ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aec0dbd3-4298-476c-91bb-2c0cebd34305", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52262025-04-30T06:26:30.416ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
52272025-04-30T06:26:30.422ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40c3919c-0064-4f90-90e4-f0c9852afb98", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52282025-04-30T06:26:30.423ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
52292025-04-30T06:26:30.430ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "72e68e3a-0045-45ff-9683-fc6914f3a2b4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52302025-04-30T06:26:30.431ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
52312025-04-30T06:26:30.437ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb8a295b-2c3a-4c53-b37b-5d97d83d7d3c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52322025-04-30T06:26:30.437ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
52332025-04-30T06:26:30.437ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
52342025-04-30T06:26:30.441ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2203599f-055e-4ff2-8e6a-8f65c6a746f9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52352025-04-30T06:26:30.442ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
52362025-04-30T06:26:30.444ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b729f9f0-d717-4a01-8f21-dcbb55cf7657", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52372025-04-30T06:26:30.444ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
52382025-04-30T06:26:30.445ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9fb57c0-5227-41ce-8188-da08b6d15d98", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52392025-04-30T06:26:30.445ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
52402025-04-30T06:26:30.447ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4de1df86-a930-4c89-a0d9-9acfd5c2a2e8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52412025-04-30T06:26:30.447ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
52422025-04-30T06:26:30.448ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1c29b5f-99af-4413-b33c-438781f08b6c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52432025-04-30T06:26:30.448ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
52442025-04-30T06:26:30.449ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43d32f8c-a2f1-4e34-b84f-7828bf190b48", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52452025-04-30T06:26:30.449ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
52462025-04-30T06:26:30.451ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d06d7e8-d227-4fa8-b65f-70cad22fdebd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52472025-04-30T06:26:30.451ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
52482025-04-30T06:26:30.453ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67200fb7-7959-4c05-b372-1a38c954aee3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52492025-04-30T06:26:30.453ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
52502025-04-30T06:26:30.455ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9496e9ea-1bbf-4e66-a2f2-cc79ce9ce794", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52512025-04-30T06:26:30.456ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
52522025-04-30T06:26:30.457ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77cd0a37-2bd3-4a0c-bf7a-e8de7d97bd5b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52532025-04-30T06:26:30.458ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
52542025-04-30T06:26:30.460ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "80386529-5ff4-43a6-abfb-6ccd706ce2bd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52552025-04-30T06:26:30.460ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
52562025-04-30T06:26:30.462ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21e41a14-f7f4-4ba7-8f40-bfd1d84e50e4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52572025-04-30T06:26:30.463ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
52582025-04-30T06:26:30.466ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5cc3b60-dcc5-4034-a607-d8671f17cd7c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52592025-04-30T06:26:30.466ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
52602025-04-30T06:26:30.468ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7764f5c1-da9d-4579-8a31-1ae3334ad37d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52612025-04-30T06:26:30.469ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
52622025-04-30T06:26:30.470ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5676442-c773-4adf-8800-64966b280c4a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52632025-04-30T06:26:30.470ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
52642025-04-30T06:26:30.473ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "afc0428b-12c7-432c-a626-8a65ae1a33d7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52652025-04-30T06:26:30.473ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
52662025-04-30T06:26:30.474ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ff3b51e-1f4a-4b4c-8e65-afec46047e3b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52672025-04-30T06:26:30.474ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
52682025-04-30T06:26:30.476ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c3fcd59-f54e-49c0-8a83-7ff4549e0dc8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52692025-04-30T06:26:30.476ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
52702025-04-30T06:26:30.479ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00265bfd-9b53-47a7-98c6-68280ecde27a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52712025-04-30T06:26:30.480ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
52722025-04-30T06:26:30.487ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "72ce6e3e-dd40-49d9-986d-c15d04e888cd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52732025-04-30T06:26:30.487ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
52742025-04-30T06:26:30.489ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52752025-04-30T06:26:30.489ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52762025-04-30T06:26:30.489ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52772025-04-30T06:26:30.489ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
52782025-04-30T06:26:30.492ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52792025-04-30T06:26:30.494ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52802025-04-30T06:26:30.505ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52812025-04-30T06:26:30.505ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52822025-04-30T06:26:30.505ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
52832025-04-30T06:26:30.506ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52842025-04-30T06:26:30.508ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e627a51f-ba6c-45f4-88c4-deb41f3f0400", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52852025-04-30T06:26:30.508ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
52862025-04-30T06:26:30.509ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52872025-04-30T06:26:30.509ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
52882025-04-30T06:26:30.509ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52892025-04-30T06:26:30.515ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04fa71c8-2371-4b27-94a3-6513e5a7fb81", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52902025-04-30T06:26:30.515ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
52912025-04-30T06:26:30.523ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df888c74-9db3-4364-a6e7-4806dffb448a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52922025-04-30T06:26:30.524ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
52932025-04-30T06:26:30.527ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "80284368-5a89-4226-8dc9-e1e2d546abcb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52942025-04-30T06:26:30.528ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
52952025-04-30T06:26:30.529ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c77af12-32e0-4d54-af73-2337fd83ebcd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52962025-04-30T06:26:30.529ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
52972025-04-30T06:26:30.531ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04e34b27-f526-4596-bc30-1b565399bc36", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
52982025-04-30T06:26:30.531ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
52992025-04-30T06:26:30.532ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6d179f8-ceab-4633-86dd-9ffed7ebc16c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53002025-04-30T06:26:30.533ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
53012025-04-30T06:26:30.534ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdd2ee10-c567-4f21-92bc-19ec057b011c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53022025-04-30T06:26:30.535ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
53032025-04-30T06:26:30.537ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a68516a8-3447-44b7-9031-e614aae54e0f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53042025-04-30T06:26:30.538ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
53052025-04-30T06:26:30.541ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a88afec0-2a82-4e7b-9244-b9d9021d91d9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53062025-04-30T06:26:30.541ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
53072025-04-30T06:26:30.545ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
53082025-04-30T06:26:30.547ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "728c2dfb-042a-46fb-856b-71d2758937ea", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53092025-04-30T06:26:30.547ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
53102025-04-30T06:26:30.549ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e09d0638-d95d-42f3-9db6-c7c396a92ca2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53112025-04-30T06:26:30.549ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
53122025-04-30T06:26:30.551ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cd82c04-9054-49ad-84c8-ec0c3fe94ce9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53132025-04-30T06:26:30.551ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
53142025-04-30T06:26:30.553ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a8a74e7-055c-4058-a81f-9ca7d4b4c959", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53152025-04-30T06:26:30.553ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
53162025-04-30T06:26:30.555ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c326d5c1-d359-4d0b-a5ba-bc30851a8107", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53172025-04-30T06:26:30.555ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
53182025-04-30T06:26:30.556ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0f73a36-aebc-46c0-8263-bb1130256f20", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53192025-04-30T06:26:30.556ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
53202025-04-30T06:26:30.557ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ede393b-303a-480f-96e6-0de4f22a9d3b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53212025-04-30T06:26:30.557ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
53222025-04-30T06:26:30.559ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60274488-959a-4059-9017-db99d463218d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53232025-04-30T06:26:30.559ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
53242025-04-30T06:26:30.560ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd543f93-2ac1-4290-aa5d-70a01fccaae3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53252025-04-30T06:26:30.560ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
53262025-04-30T06:26:30.560ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53272025-04-30T06:26:30.560ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53282025-04-30T06:26:30.560ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53292025-04-30T06:26:30.562ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53302025-04-30T06:26:30.564ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53312025-04-30T06:26:30.570ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53322025-04-30T06:26:30.570ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
53332025-04-30T06:26:30.570ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53342025-04-30T06:26:30.570ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53352025-04-30T06:26:30.570ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53362025-04-30T06:26:30.570ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53372025-04-30T06:26:30.573ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53382025-04-30T06:26:30.579ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53392025-04-30T06:26:30.580ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53402025-04-30T06:26:30.580ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53412025-04-30T06:26:30.580ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53422025-04-30T06:26:30.648ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
53432025-04-30T06:26:30.648ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
53442025-04-30T06:26:30.648ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53452025-04-30T06:26:30.648ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53462025-04-30T06:26:30.649ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53472025-04-30T06:26:30.649ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
53482025-04-30T06:26:30.650ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53492025-04-30T06:26:30.650ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:34096 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
53502025-04-30T06:26:30.652ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:34096 req_id = 42c9cec1-4304-4a08-bdb4-c0956d043b5d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
53512025-04-30T06:26:30.652ZINFOtest_disk_expunge (dropshot): request completed latency_us = 613 local_addr = [::1]:44559 method = GET remote_addr = [::1]:34096 req_id = 42c9cec1-4304-4a08-bdb4-c0956d043b5d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
53522025-04-30T06:26:30.653ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "42c9cec1-4304-4a08-bdb4-c0956d043b5d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
53532025-04-30T06:26:30.655ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53542025-04-30T06:26:30.656ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53552025-04-30T06:26:30.656ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
53562025-04-30T06:26:30.659ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53572025-04-30T06:26:30.659ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53582025-04-30T06:26:30.659ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53592025-04-30T06:26:30.659ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53602025-04-30T06:26:30.660ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53612025-04-30T06:26:30.661ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53622025-04-30T06:26:30.667ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53632025-04-30T06:26:30.667ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53642025-04-30T06:26:30.668ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
53652025-04-30T06:26:30.668ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53662025-04-30T06:26:30.671ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53672025-04-30T06:26:30.678ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53682025-04-30T06:26:30.679ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53692025-04-30T06:26:30.679ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53702025-04-30T06:26:30.680ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53712025-04-30T06:26:30.697ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53722025-04-30T06:26:30.698ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
53732025-04-30T06:26:30.698ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53742025-04-30T06:26:30.765ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
53752025-04-30T06:26:30.766ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
53762025-04-30T06:26:30.766ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 449.076436ms iteration = 13 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
53772025-04-30T06:26:30.766ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 14 reason = Signaled
53782025-04-30T06:26:30.766ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53792025-04-30T06:26:30.766ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53802025-04-30T06:26:30.766ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53812025-04-30T06:26:30.766ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53822025-04-30T06:26:30.768ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53832025-04-30T06:26:30.771ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53842025-04-30T06:26:30.777ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53852025-04-30T06:26:30.777ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53862025-04-30T06:26:30.777ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53872025-04-30T06:26:30.777ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53882025-04-30T06:26:30.784ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
53892025-04-30T06:26:30.784ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
53902025-04-30T06:26:30.785ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cb f5 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49879 req_id = 286514df-69d6-49c4-a454-36efaa219894
53912025-04-30T06:26:30.785ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52213,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49879 req_id = 286514df-69d6-49c4-a454-36efaa219894
53922025-04-30T06:26:30.785ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
53932025-04-30T06:26:30.786ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53942025-04-30T06:26:30.786ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49879 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 286514df-69d6-49c4-a454-36efaa219894
53952025-04-30T06:26:30.787ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3255, tv_nsec: 586017279 } })
53962025-04-30T06:26:30.787ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
53972025-04-30T06:26:30.787ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
53982025-04-30T06:26:30.788ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d5 1f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52693 req_id = 95c6d465-5db8-4e37-b160-981b609b89ea
53992025-04-30T06:26:30.788ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54559,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52693 req_id = 95c6d465-5db8-4e37-b160-981b609b89ea
54002025-04-30T06:26:30.788ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
54012025-04-30T06:26:30.788ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52693 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 95c6d465-5db8-4e37-b160-981b609b89ea
54022025-04-30T06:26:30.869ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
54032025-04-30T06:26:30.871ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54042025-04-30T06:26:30.872ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54052025-04-30T06:26:30.872ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54062025-04-30T06:26:30.872ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
54072025-04-30T06:26:30.873ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54082025-04-30T06:26:30.873ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 40 35 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c @5..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59091 req_id = 9bf9fd2a-62a6-44a2-a6d6-01cc25b4083a
54092025-04-30T06:26:30.873ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16437,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59091 req_id = 9bf9fd2a-62a6-44a2-a6d6-01cc25b4083a
54102025-04-30T06:26:30.874ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
54112025-04-30T06:26:30.874ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16437, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59091 req_id = 9bf9fd2a-62a6-44a2-a6d6-01cc25b4083a
54122025-04-30T06:26:30.874ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
54132025-04-30T06:26:30.875ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
54142025-04-30T06:26:30.875ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
54152025-04-30T06:26:30.875ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
54162025-04-30T06:26:30.876ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
54172025-04-30T06:26:30.876ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54182025-04-30T06:26:30.888ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54192025-04-30T06:26:30.891ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54202025-04-30T06:26:30.892ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
54212025-04-30T06:26:30.892ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54222025-04-30T06:26:30.893ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 0894fc1b-7c4b-4cb2-9b71-cc9ea78f3866 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
54232025-04-30T06:26:30.893ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 0894fc1b-7c4b-4cb2-9b71-cc9ea78f3866 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
54242025-04-30T06:26:30.893ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 0894fc1b-7c4b-4cb2-9b71-cc9ea78f3866 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
54252025-04-30T06:26:30.894ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 0894fc1b-7c4b-4cb2-9b71-cc9ea78f3866 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
54262025-04-30T06:26:30.897ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "150c27be-f346-4919-880e-301400ed7937", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54272025-04-30T06:26:30.897ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
54282025-04-30T06:26:30.898ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54292025-04-30T06:26:30.898ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
54302025-04-30T06:26:30.899ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54312025-04-30T06:26:30.900ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 7563 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 0894fc1b-7c4b-4cb2-9b71-cc9ea78f3866 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
54322025-04-30T06:26:30.902ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "0894fc1b-7c4b-4cb2-9b71-cc9ea78f3866", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54332025-04-30T06:26:30.903ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
54342025-04-30T06:26:30.903ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d34ed12a-862e-4cdb-a151-c03ca86a516d uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
54352025-04-30T06:26:30.904ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d34ed12a-862e-4cdb-a151-c03ca86a516d resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
54362025-04-30T06:26:30.904ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d34ed12a-862e-4cdb-a151-c03ca86a516d roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
54372025-04-30T06:26:30.905ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d34ed12a-862e-4cdb-a151-c03ca86a516d resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
54382025-04-30T06:26:30.906ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "400c1c2e-b9cc-4a6e-b8e7-d67eda4d5cc7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54392025-04-30T06:26:30.908ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
54402025-04-30T06:26:30.909ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 5518 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d34ed12a-862e-4cdb-a151-c03ca86a516d response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
54412025-04-30T06:26:30.910ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "d34ed12a-862e-4cdb-a151-c03ca86a516d", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54422025-04-30T06:26:30.911ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
54432025-04-30T06:26:30.911ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
54442025-04-30T06:26:30.911ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
54452025-04-30T06:26:30.916ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be8c52f8-8b17-4084-b02c-decf766b878a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54462025-04-30T06:26:30.916ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
54472025-04-30T06:26:30.924ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "364f5462-2876-431d-85f8-d4b0502bfc3c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54482025-04-30T06:26:30.924ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
54492025-04-30T06:26:30.929ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0941ccc1-f461-4ffc-84f8-ef72d8d85515", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54502025-04-30T06:26:30.929ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
54512025-04-30T06:26:30.933ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
54522025-04-30T06:26:30.936ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0de91dc-1078-4e9a-9112-7fa1e3e69874", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54532025-04-30T06:26:30.936ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
54542025-04-30T06:26:30.941ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdfac345-c2f4-4d43-bed3-2818d2a82694", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54552025-04-30T06:26:30.941ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
54562025-04-30T06:26:30.944ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c52db78b-e717-41d8-85b8-3143cd89fe29", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54572025-04-30T06:26:30.944ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
54582025-04-30T06:26:30.945ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "410eff02-0e53-4ba8-b00b-db08e9d7382b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54592025-04-30T06:26:30.945ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
54602025-04-30T06:26:30.946ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d40e6df-d594-4b98-84b7-699f0ca7f891", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54612025-04-30T06:26:30.946ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
54622025-04-30T06:26:30.947ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b8a7424-7f97-4786-9e27-0506fe6a6eab", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54632025-04-30T06:26:30.947ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
54642025-04-30T06:26:30.948ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "779a131d-38fa-4291-a72d-a49acdf02692", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54652025-04-30T06:26:30.948ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
54662025-04-30T06:26:30.950ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "806f988a-9005-45b9-98e8-b92e8fbd73a3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54672025-04-30T06:26:30.950ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
54682025-04-30T06:26:30.957ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
54692025-04-30T06:26:30.957ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
54702025-04-30T06:26:30.957ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
54712025-04-30T06:26:30.959ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
54722025-04-30T06:26:30.961ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "766d97b0-ebe2-424f-995d-a88ad93861ed", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54732025-04-30T06:26:30.961ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
54742025-04-30T06:26:30.965ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
54752025-04-30T06:26:30.965ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
54762025-04-30T06:26:30.970ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
54772025-04-30T06:26:30.970ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
54782025-04-30T06:26:30.971ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46cc21ca-a6f8-4744-9b55-2ebc313df047", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54792025-04-30T06:26:30.971ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
54802025-04-30T06:26:30.973ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac299225-ec2f-4fdc-87a7-086a748dd3d0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54812025-04-30T06:26:30.973ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
54822025-04-30T06:26:30.975ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "309a4891-10c0-4426-bc46-1952e6780d24", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54832025-04-30T06:26:30.975ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
54842025-04-30T06:26:30.977ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b33f9361-4816-474c-81f9-18532fb722d0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54852025-04-30T06:26:30.978ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
54862025-04-30T06:26:30.980ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5baa3c76-c2a8-4eea-b822-4dec8288d756", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54872025-04-30T06:26:30.980ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
54882025-04-30T06:26:30.981ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "993c63d5-88ba-4a85-acbd-accc7d77319d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54892025-04-30T06:26:30.981ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
54902025-04-30T06:26:30.984ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ccea8d7-ec39-4ab3-af4b-399b07aee39f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54912025-04-30T06:26:30.984ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
54922025-04-30T06:26:30.985ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54932025-04-30T06:26:30.985ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54942025-04-30T06:26:30.985ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54952025-04-30T06:26:30.985ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
54962025-04-30T06:26:30.987ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54972025-04-30T06:26:30.988ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9718c33-ab5f-4b62-99d6-ccb40f5b3d00", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
54982025-04-30T06:26:30.988ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
54992025-04-30T06:26:30.990ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55002025-04-30T06:26:30.996ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55012025-04-30T06:26:30.996ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55022025-04-30T06:26:30.996ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
55032025-04-30T06:26:30.997ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55042025-04-30T06:26:30.998ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97016da8-8637-4839-b96c-772f150eeb0b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
55052025-04-30T06:26:30.998ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
55062025-04-30T06:26:31.018ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55072025-04-30T06:26:31.018ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
55082025-04-30T06:26:31.020ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55092025-04-30T06:26:31.036ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "948c36d4-762c-4e17-9509-fc4faa714542", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
55102025-04-30T06:26:31.037ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
55112025-04-30T06:26:31.043ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55122025-04-30T06:26:31.045ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55132025-04-30T06:26:31.059ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55142025-04-30T06:26:31.060ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55152025-04-30T06:26:31.060ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55162025-04-30T06:26:31.060ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
55172025-04-30T06:26:31.062ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
55182025-04-30T06:26:31.079ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e91c75a2-f4be-4a50-8b61-8c40a1249d58", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:30 GMT"} })
55192025-04-30T06:26:31.079ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
55202025-04-30T06:26:31.083ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
55212025-04-30T06:26:31.084ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6632966f-3199-4512-8542-af4eb696a2e5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55222025-04-30T06:26:31.084ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
55232025-04-30T06:26:31.086ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e04c334f-2372-4e4a-bc94-f080820c90e6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55242025-04-30T06:26:31.086ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
55252025-04-30T06:26:31.088ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "142ed19a-a563-45f8-af20-e488ac2c806a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55262025-04-30T06:26:31.088ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
55272025-04-30T06:26:31.091ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6eb0c978-94df-4589-a761-d34d7d4905df", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55282025-04-30T06:26:31.091ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
55292025-04-30T06:26:31.093ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d86faeda-23d2-4fac-a282-70f61fd3f4f4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55302025-04-30T06:26:31.093ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
55312025-04-30T06:26:31.094ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "500ade2c-2a06-420f-a4aa-91aff4b08601", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55322025-04-30T06:26:31.094ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
55332025-04-30T06:26:31.095ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92dc7194-6616-481e-84fe-17b7766d3ae0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55342025-04-30T06:26:31.095ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
55352025-04-30T06:26:31.097ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd56f39e-70c2-4188-8709-4a258d7b6316", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55362025-04-30T06:26:31.097ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
55372025-04-30T06:26:31.100ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "142f1f99-6c19-45b6-9794-18e1d633f7d7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55382025-04-30T06:26:31.100ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
55392025-04-30T06:26:31.101ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9578d6bf-acf2-4288-b704-f97f055e37ab", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55402025-04-30T06:26:31.101ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
55412025-04-30T06:26:31.103ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ea0f93c-afdb-4c63-84cc-b4b3b0efb1b8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55422025-04-30T06:26:31.103ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
55432025-04-30T06:26:31.104ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "205961b6-c63e-4b5d-b3af-942c1ab8ae0e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55442025-04-30T06:26:31.104ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
55452025-04-30T06:26:31.106ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f811caf0-e4dd-412d-b961-7f636cecd8d2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55462025-04-30T06:26:31.106ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
55472025-04-30T06:26:31.109ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "652e1af1-fe8a-4762-ad06-5ff384d322cc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55482025-04-30T06:26:31.109ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
55492025-04-30T06:26:31.112ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f04da4b-c40c-4e08-92fa-9a422f074e09", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55502025-04-30T06:26:31.112ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
55512025-04-30T06:26:31.116ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1798694-ffdc-4bd3-b05d-b4c7d2bc4b62", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55522025-04-30T06:26:31.116ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
55532025-04-30T06:26:31.117ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d933fb03-cfa8-48b2-8e90-7bb813fe3cf1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55542025-04-30T06:26:31.117ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
55552025-04-30T06:26:31.118ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65c67322-64a3-44c4-9b70-82b60e7e1710", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55562025-04-30T06:26:31.118ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
55572025-04-30T06:26:31.120ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77199438-8bfa-4e1f-b4cb-d8aeb9992116", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55582025-04-30T06:26:31.120ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
55592025-04-30T06:26:31.120ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcedea19-78c5-43fb-a13e-1836cbbee261", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55602025-04-30T06:26:31.120ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55612025-04-30T06:26:31.120ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55622025-04-30T06:26:31.121ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55632025-04-30T06:26:31.121ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55642025-04-30T06:26:31.121ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55652025-04-30T06:26:31.127ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55662025-04-30T06:26:31.134ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55672025-04-30T06:26:31.134ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55682025-04-30T06:26:31.135ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55692025-04-30T06:26:31.135ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55702025-04-30T06:26:31.135ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55712025-04-30T06:26:31.135ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55722025-04-30T06:26:31.141ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55732025-04-30T06:26:31.141ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55742025-04-30T06:26:31.141ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55752025-04-30T06:26:31.141ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
55762025-04-30T06:26:31.142ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55772025-04-30T06:26:31.159ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55782025-04-30T06:26:31.183ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55792025-04-30T06:26:31.183ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55802025-04-30T06:26:31.183ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55812025-04-30T06:26:31.183ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55822025-04-30T06:26:31.185ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55832025-04-30T06:26:31.196ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55842025-04-30T06:26:31.196ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55852025-04-30T06:26:31.197ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
55862025-04-30T06:26:31.198ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55872025-04-30T06:26:31.202ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55882025-04-30T06:26:31.202ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
55892025-04-30T06:26:31.202ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55902025-04-30T06:26:31.262ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
55912025-04-30T06:26:31.262ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
55922025-04-30T06:26:31.262ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:37995 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
55932025-04-30T06:26:31.263ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:37995 req_id = 92bd1d87-5c3e-434c-a1c5-8f26d64de164 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55942025-04-30T06:26:31.264ZINFOtest_disk_expunge (dropshot): request completed latency_us = 568 local_addr = [::1]:44559 method = GET remote_addr = [::1]:37995 req_id = 92bd1d87-5c3e-434c-a1c5-8f26d64de164 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55952025-04-30T06:26:31.264ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "92bd1d87-5c3e-434c-a1c5-8f26d64de164", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
55962025-04-30T06:26:31.265ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55972025-04-30T06:26:31.265ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55982025-04-30T06:26:31.265ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
55992025-04-30T06:26:31.267ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
56002025-04-30T06:26:31.267ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56012025-04-30T06:26:31.267ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56022025-04-30T06:26:31.267ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56032025-04-30T06:26:31.268ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56042025-04-30T06:26:31.272ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56052025-04-30T06:26:31.279ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
56062025-04-30T06:26:31.280ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56072025-04-30T06:26:31.280ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56082025-04-30T06:26:31.281ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56092025-04-30T06:26:31.321ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
56102025-04-30T06:26:31.332ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
56112025-04-30T06:26:31.332ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
56122025-04-30T06:26:31.332ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 566.28218ms iteration = 14 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
56132025-04-30T06:26:31.332ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 15 reason = Signaled
56142025-04-30T06:26:31.332ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
56152025-04-30T06:26:31.332ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56162025-04-30T06:26:31.332ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56172025-04-30T06:26:31.332ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56182025-04-30T06:26:31.333ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56192025-04-30T06:26:31.343ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56202025-04-30T06:26:31.348ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
56212025-04-30T06:26:31.348ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56222025-04-30T06:26:31.348ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56232025-04-30T06:26:31.348ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56242025-04-30T06:26:31.353ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
56252025-04-30T06:26:31.353ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
56262025-04-30T06:26:31.353ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: d5 5d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .]..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50907 req_id = 40d3b31c-6132-4569-a9ba-ca1c4f99bf90
56272025-04-30T06:26:31.354ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54621,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50907 req_id = 40d3b31c-6132-4569-a9ba-ca1c4f99bf90
56282025-04-30T06:26:31.354ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
56292025-04-30T06:26:31.354ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
56302025-04-30T06:26:31.354ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50907 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 40d3b31c-6132-4569-a9ba-ca1c4f99bf90
56312025-04-30T06:26:31.354ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3256, tv_nsec: 153731182 } })
56322025-04-30T06:26:31.354ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
56332025-04-30T06:26:31.355ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 96 04 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60123 req_id = cfba9979-47cd-46b9-99e2-05badaa5dfd6
56342025-04-30T06:26:31.355ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38404,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60123 req_id = cfba9979-47cd-46b9-99e2-05badaa5dfd6
56352025-04-30T06:26:31.355ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
56362025-04-30T06:26:31.356ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60123 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cfba9979-47cd-46b9-99e2-05badaa5dfd6
56372025-04-30T06:26:31.424ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
56382025-04-30T06:26:31.425ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56392025-04-30T06:26:31.425ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56402025-04-30T06:26:31.425ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56412025-04-30T06:26:31.425ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
56422025-04-30T06:26:31.426ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56432025-04-30T06:26:31.426ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9f 98 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60987 req_id = a348eea0-4277-485c-a635-ea4efb42c7f6
56442025-04-30T06:26:31.426ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40856,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60987 req_id = a348eea0-4277-485c-a635-ea4efb42c7f6
56452025-04-30T06:26:31.426ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
56462025-04-30T06:26:31.426ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40856, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60987 req_id = a348eea0-4277-485c-a635-ea4efb42c7f6
56472025-04-30T06:26:31.427ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
56482025-04-30T06:26:31.427ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
56492025-04-30T06:26:31.427ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
56502025-04-30T06:26:31.428ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ea828b8-8506-4e3b-a4c8-27b05562493c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56512025-04-30T06:26:31.429ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
56522025-04-30T06:26:31.429ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56532025-04-30T06:26:31.434ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56542025-04-30T06:26:31.434ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56552025-04-30T06:26:31.435ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
56562025-04-30T06:26:31.435ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56572025-04-30T06:26:31.437ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d852aa9-5a36-4f3a-8537-6b5963a341b6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56582025-04-30T06:26:31.438ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
56592025-04-30T06:26:31.448ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56602025-04-30T06:26:31.448ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
56612025-04-30T06:26:31.449ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56622025-04-30T06:26:31.470ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a80fed3-0f0d-4d36-9e7d-6005f2014bec", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56632025-04-30T06:26:31.470ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
56642025-04-30T06:26:31.473ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74dce2ac-abe7-4e9b-a701-eeee42eaf273", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56652025-04-30T06:26:31.473ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
56662025-04-30T06:26:31.476ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e503fa0-a8a1-43d3-bb83-cc8e98fd580d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56672025-04-30T06:26:31.476ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
56682025-04-30T06:26:31.479ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca56a53c-3ac5-4710-b916-23b2b54aeece", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56692025-04-30T06:26:31.480ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
56702025-04-30T06:26:31.482ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b95eb0ba-5cb1-4d51-96d4-581bb517d8f0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56712025-04-30T06:26:31.482ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
56722025-04-30T06:26:31.487ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3d39bec-4f43-4d45-8141-f0625e8ce8c0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56732025-04-30T06:26:31.487ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
56742025-04-30T06:26:31.491ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c5addab-46ac-42b9-9615-74acd55e1a15", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56752025-04-30T06:26:31.492ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
56762025-04-30T06:26:31.514ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5adee74-3dcb-474c-a139-ec15388ca335", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56772025-04-30T06:26:31.514ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
56782025-04-30T06:26:31.515ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
56792025-04-30T06:26:31.518ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ca121aa-b0bd-4fc6-a837-31be7f8f2a61", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56802025-04-30T06:26:31.518ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
56812025-04-30T06:26:31.520ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dee4244-2202-4c9a-9077-9ede457e82fb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56822025-04-30T06:26:31.520ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
56832025-04-30T06:26:31.522ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ca83e62-b587-4ced-88e1-e7a306503455", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56842025-04-30T06:26:31.522ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
56852025-04-30T06:26:31.527ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eba9f1d5-5d95-49a0-8ac6-ac948bb96c98", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56862025-04-30T06:26:31.527ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
56872025-04-30T06:26:31.542ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18a4f6b3-1d44-48fa-bb3c-e6d2d3ece747", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56882025-04-30T06:26:31.542ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
56892025-04-30T06:26:31.546ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b1d2caa-089b-4169-a5db-4ec5a207fba4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56902025-04-30T06:26:31.546ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
56912025-04-30T06:26:31.553ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "450d61a0-1ec7-47f7-9f0c-0dc59f153ea7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
56922025-04-30T06:26:31.565ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
56932025-04-30T06:26:31.565ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56942025-04-30T06:26:31.566ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56952025-04-30T06:26:31.566ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56962025-04-30T06:26:31.566ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
56972025-04-30T06:26:31.566ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56982025-04-30T06:26:31.576ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56992025-04-30T06:26:31.593ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57002025-04-30T06:26:31.593ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57012025-04-30T06:26:31.593ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
57022025-04-30T06:26:31.594ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57032025-04-30T06:26:31.594ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e443e0b9-fe0a-46e6-a8d4-49abdd3fb5e4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57042025-04-30T06:26:31.595ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
57052025-04-30T06:26:31.598ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f289ad84-203a-4a96-ad80-6701078e0399", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57062025-04-30T06:26:31.598ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
57072025-04-30T06:26:31.600ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57082025-04-30T06:26:31.600ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
57092025-04-30T06:26:31.601ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57102025-04-30T06:26:31.602ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d170939-34f3-443d-bd90-2e912d1520b0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57112025-04-30T06:26:31.614ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
57122025-04-30T06:26:31.621ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bf960a0-29d7-4694-aa65-ef170cc867a6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57132025-04-30T06:26:31.622ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
57142025-04-30T06:26:31.625ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90a58dd5-6b0b-46dd-8c92-b575e40627c8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57152025-04-30T06:26:31.625ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
57162025-04-30T06:26:31.626ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef743e89-5b1b-4a41-b835-73cadf61a3e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57172025-04-30T06:26:31.626ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
57182025-04-30T06:26:31.631ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79385d9f-4bab-45df-af50-ce6a54f6fb02", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57192025-04-30T06:26:31.631ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
57202025-04-30T06:26:31.635ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69918612-b306-47e1-b565-fea7a809628d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57212025-04-30T06:26:31.635ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
57222025-04-30T06:26:31.639ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5652ec0a-9ee2-4265-961b-327cf0bdb1fe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57232025-04-30T06:26:31.639ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
57242025-04-30T06:26:31.644ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cc38fb1-807e-4988-8046-cb2d525363b5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57252025-04-30T06:26:31.644ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
57262025-04-30T06:26:31.654ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22956bd0-f753-427b-9b70-ceee02b69d8c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57272025-04-30T06:26:31.654ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
57282025-04-30T06:26:31.655ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
57292025-04-30T06:26:31.657ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ae93688-551c-49dd-a53c-d16c86489f78", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57302025-04-30T06:26:31.657ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
57312025-04-30T06:26:31.661ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee71e599-e75f-4154-97e1-631b7f198321", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57322025-04-30T06:26:31.661ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
57332025-04-30T06:26:31.662ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9193a32b-2f6d-4807-82b5-f450d34a4f8d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57342025-04-30T06:26:31.663ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
57352025-04-30T06:26:31.663ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "623e2797-2c00-4e81-9cad-aa831287ad62", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57362025-04-30T06:26:31.664ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
57372025-04-30T06:26:31.665ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5e80788-d8e0-4080-970f-d8e8c66afd25", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57382025-04-30T06:26:31.665ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
57392025-04-30T06:26:31.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "191348a6-4327-4f92-a98c-79f970914b96", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57402025-04-30T06:26:31.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
57412025-04-30T06:26:31.667ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21445055-80f1-45f6-9fb4-d505b06747ae", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57422025-04-30T06:26:31.667ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
57432025-04-30T06:26:31.669ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bf5a11b-1703-454c-a65b-4d209ff6eb1e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57442025-04-30T06:26:31.669ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
57452025-04-30T06:26:31.670ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e6151dc-373c-4360-9128-5260f28fa500", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57462025-04-30T06:26:31.671ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
57472025-04-30T06:26:31.672ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31803790-138b-4dcb-9956-494670e2b6ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57482025-04-30T06:26:31.673ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
57492025-04-30T06:26:31.674ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4bb1b8a-17b5-4877-971a-59efa96e8ab1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57502025-04-30T06:26:31.674ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
57512025-04-30T06:26:31.676ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d3220a3-7aaf-4f8e-ab5a-2607e5544eb1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57522025-04-30T06:26:31.676ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
57532025-04-30T06:26:31.678ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af634df4-1314-440b-a232-67af886f33c7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57542025-04-30T06:26:31.678ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
57552025-04-30T06:26:31.679ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55822744-a77a-48e0-9b9d-69979dc8d5f0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57562025-04-30T06:26:31.680ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
57572025-04-30T06:26:31.686ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a5101b1-079a-45af-a182-d6e693f438bd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57582025-04-30T06:26:31.686ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
57592025-04-30T06:26:31.686ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1b98eb6-c46e-4133-8347-ff03f0051d64", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57602025-04-30T06:26:31.686ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
57612025-04-30T06:26:31.695ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "216c821e-8d3d-47c4-8ae3-23c3f0a05b6f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57622025-04-30T06:26:31.696ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57632025-04-30T06:26:31.696ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57642025-04-30T06:26:31.697ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57652025-04-30T06:26:31.697ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57662025-04-30T06:26:31.697ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57672025-04-30T06:26:31.726ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57682025-04-30T06:26:31.726ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57692025-04-30T06:26:31.726ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57702025-04-30T06:26:31.726ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
57712025-04-30T06:26:31.728ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57722025-04-30T06:26:31.730ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57732025-04-30T06:26:31.739ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57742025-04-30T06:26:31.739ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57752025-04-30T06:26:31.739ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57762025-04-30T06:26:31.739ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57772025-04-30T06:26:31.739ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57782025-04-30T06:26:31.740ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57792025-04-30T06:26:31.744ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57802025-04-30T06:26:31.759ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57812025-04-30T06:26:31.759ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57822025-04-30T06:26:31.759ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
57832025-04-30T06:26:31.760ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57842025-04-30T06:26:31.760ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57852025-04-30T06:26:31.767ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57862025-04-30T06:26:31.767ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57872025-04-30T06:26:31.767ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57882025-04-30T06:26:31.768ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57892025-04-30T06:26:31.788ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57902025-04-30T06:26:31.788ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
57912025-04-30T06:26:31.789ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57922025-04-30T06:26:31.873ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
57932025-04-30T06:26:31.874ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
57942025-04-30T06:26:31.874ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:34732 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
57952025-04-30T06:26:31.877ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:34732 req_id = e22dc4e4-6670-4d54-b355-248c95bad4c2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57962025-04-30T06:26:31.877ZINFOtest_disk_expunge (dropshot): request completed latency_us = 477 local_addr = [::1]:44559 method = GET remote_addr = [::1]:34732 req_id = e22dc4e4-6670-4d54-b355-248c95bad4c2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57972025-04-30T06:26:31.878ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e22dc4e4-6670-4d54-b355-248c95bad4c2", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:31 GMT"} })
57982025-04-30T06:26:31.883ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57992025-04-30T06:26:31.884ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
58002025-04-30T06:26:31.884ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
58012025-04-30T06:26:31.885ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
58022025-04-30T06:26:31.885ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58032025-04-30T06:26:31.885ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58042025-04-30T06:26:31.885ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58052025-04-30T06:26:31.886ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58062025-04-30T06:26:31.892ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58072025-04-30T06:26:31.907ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
58082025-04-30T06:26:31.907ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58092025-04-30T06:26:31.907ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58102025-04-30T06:26:31.908ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58112025-04-30T06:26:31.962ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58122025-04-30T06:26:31.962ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58132025-04-30T06:26:31.962ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
58142025-04-30T06:26:31.964ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
58152025-04-30T06:26:31.966ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
58162025-04-30T06:26:31.972ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
58172025-04-30T06:26:31.972ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
58182025-04-30T06:26:31.975ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
58192025-04-30T06:26:31.976ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
58202025-04-30T06:26:31.976ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 643.764989ms iteration = 15 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
58212025-04-30T06:26:31.976ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 16 reason = Signaled
58222025-04-30T06:26:31.976ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
58232025-04-30T06:26:31.976ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58242025-04-30T06:26:31.976ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58252025-04-30T06:26:31.976ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58262025-04-30T06:26:31.977ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58272025-04-30T06:26:31.978ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
58282025-04-30T06:26:31.978ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
58292025-04-30T06:26:31.981ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58302025-04-30T06:26:31.988ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
58312025-04-30T06:26:31.988ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
58322025-04-30T06:26:31.988ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
58332025-04-30T06:26:31.989ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
58342025-04-30T06:26:32.018ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58352025-04-30T06:26:32.018ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58362025-04-30T06:26:32.018ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58372025-04-30T06:26:32.019ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
58382025-04-30T06:26:32.020ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58392025-04-30T06:26:32.020ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
58402025-04-30T06:26:32.020ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
58412025-04-30T06:26:32.028ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e2 e5 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49555 req_id = 7f171b20-ee64-45df-ac9c-9814a340a6a1
58422025-04-30T06:26:32.028ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58085,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49555 req_id = 7f171b20-ee64-45df-ac9c-9814a340a6a1
58432025-04-30T06:26:32.029ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
58442025-04-30T06:26:32.029ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
58452025-04-30T06:26:32.029ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49555 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7f171b20-ee64-45df-ac9c-9814a340a6a1
58462025-04-30T06:26:32.030ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58472025-04-30T06:26:32.045ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58482025-04-30T06:26:32.045ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58492025-04-30T06:26:32.045ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
58502025-04-30T06:26:32.046ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58512025-04-30T06:26:32.048ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3256, tv_nsec: 847287147 } })
58522025-04-30T06:26:32.048ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
58532025-04-30T06:26:32.057ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58542025-04-30T06:26:32.057ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58552025-04-30T06:26:32.063ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7b b8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e {...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59728 req_id = 76180922-b88c-43f7-b456-b64f1bbc8ecf
58562025-04-30T06:26:32.063ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31672,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59728 req_id = 76180922-b88c-43f7-b456-b64f1bbc8ecf
58572025-04-30T06:26:32.063ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
58582025-04-30T06:26:32.063ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59728 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 76180922-b88c-43f7-b456-b64f1bbc8ecf
58592025-04-30T06:26:32.065ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58602025-04-30T06:26:32.065ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
58612025-04-30T06:26:32.066ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58622025-04-30T06:26:32.131ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
58632025-04-30T06:26:32.132ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58642025-04-30T06:26:32.133ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58652025-04-30T06:26:32.133ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58662025-04-30T06:26:32.134ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
58672025-04-30T06:26:32.134ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
58682025-04-30T06:26:32.149ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b2 a3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51150 req_id = 429559ac-80a1-4474-85f6-29df535bc97e
58692025-04-30T06:26:32.149ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45731,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51150 req_id = 429559ac-80a1-4474-85f6-29df535bc97e
58702025-04-30T06:26:32.149ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
58712025-04-30T06:26:32.149ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45731, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51150 req_id = 429559ac-80a1-4474-85f6-29df535bc97e
58722025-04-30T06:26:32.150ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
58732025-04-30T06:26:32.150ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
58742025-04-30T06:26:32.150ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
58752025-04-30T06:26:32.158ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "725ff80f-dce7-4e14-9a37-1f4c951126b9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
58762025-04-30T06:26:32.159ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
58772025-04-30T06:26:32.176ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15ca1cf7-6503-4a2e-ba63-36a7c2de1a30", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
58782025-04-30T06:26:32.176ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
58792025-04-30T06:26:32.179ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "47089cf7-4515-424c-8945-9f3bcfeea999", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
58802025-04-30T06:26:32.179ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
58812025-04-30T06:26:32.183ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cc90cfd-0912-42ce-ae66-e0797f842bc2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
58822025-04-30T06:26:32.183ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
58832025-04-30T06:26:32.186ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cbaacff-0891-43dd-b059-450b800f831c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
58842025-04-30T06:26:32.186ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
58852025-04-30T06:26:32.211ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
58862025-04-30T06:26:32.212ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a84d3de3-6888-425a-930b-2945dfa9bcee", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
58872025-04-30T06:26:32.212ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
58882025-04-30T06:26:32.214ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1754e4a-b6c4-40e7-b327-185aba281d2a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
58892025-04-30T06:26:32.214ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
58902025-04-30T06:26:32.219ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a91308d-5805-4718-bc87-034db4ff6fff", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
58912025-04-30T06:26:32.219ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
58922025-04-30T06:26:32.222ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c51ac9a-f44b-4179-a02e-eebf123f8b97", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
58932025-04-30T06:26:32.223ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
58942025-04-30T06:26:32.225ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5440c2d-9376-434f-8a50-2844ddb46027", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
58952025-04-30T06:26:32.225ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
58962025-04-30T06:26:32.226ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "444e3ec5-5056-44b2-a51a-4f9af79de137", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
58972025-04-30T06:26:32.226ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
58982025-04-30T06:26:32.228ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2841945c-a720-461a-aaf3-0fefc715e834", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
58992025-04-30T06:26:32.229ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
59002025-04-30T06:26:32.232ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c44f2ca8-af90-439c-bb95-69936ba1cf62", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59012025-04-30T06:26:32.232ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
59022025-04-30T06:26:32.235ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "42697c80-5dd3-488f-9fff-fb72f8846e64", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59032025-04-30T06:26:32.242ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
59042025-04-30T06:26:32.246ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fcb5887-6549-4e90-a3d0-6edce20602cf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59052025-04-30T06:26:32.246ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
59062025-04-30T06:26:32.247ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96ad6594-3aca-4acb-a289-2f4dd453be60", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59072025-04-30T06:26:32.248ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
59082025-04-30T06:26:32.249ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6928c7cf-a182-461b-b6c2-4e3e2b8c0caa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59092025-04-30T06:26:32.250ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
59102025-04-30T06:26:32.253ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d72e16aa-bd39-465a-a3c9-6e25e7cf9404", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59112025-04-30T06:26:32.253ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
59122025-04-30T06:26:32.264ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59132025-04-30T06:26:32.264ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59142025-04-30T06:26:32.264ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59152025-04-30T06:26:32.265ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
59162025-04-30T06:26:32.266ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59172025-04-30T06:26:32.268ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59182025-04-30T06:26:32.275ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59192025-04-30T06:26:32.275ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59202025-04-30T06:26:32.276ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
59212025-04-30T06:26:32.276ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59222025-04-30T06:26:32.278ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e207e71-d00f-49cb-9a35-b3ec07ba004a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59232025-04-30T06:26:32.278ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
59242025-04-30T06:26:32.279ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59252025-04-30T06:26:32.279ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
59262025-04-30T06:26:32.279ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59272025-04-30T06:26:32.281ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f38382a6-98a4-45b5-9833-506173f41934", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59282025-04-30T06:26:32.281ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
59292025-04-30T06:26:32.283ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba9ca4cc-426a-49ca-886a-fb4a7c3c3b21", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59302025-04-30T06:26:32.283ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
59312025-04-30T06:26:32.285ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfed94d0-54bd-425c-a836-c1771ba6d36d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59322025-04-30T06:26:32.285ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
59332025-04-30T06:26:32.289ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea9f5b22-baac-4088-83cf-3be0a1d85c46", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59342025-04-30T06:26:32.289ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
59352025-04-30T06:26:32.295ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bf6f8e5-3d8c-46e3-94fa-f3e63f32be15", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59362025-04-30T06:26:32.296ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
59372025-04-30T06:26:32.298ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1361b067-80d9-4ca7-9745-9f244f188103", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59382025-04-30T06:26:32.298ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
59392025-04-30T06:26:32.309ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49a39789-1073-4d01-8419-07988427034c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59402025-04-30T06:26:32.310ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
59412025-04-30T06:26:32.313ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11a1d71e-f16d-4ad0-852c-0ce9aea346ce", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59422025-04-30T06:26:32.314ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
59432025-04-30T06:26:32.322ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e58f7094-3370-4124-8c09-069f245929ea", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59442025-04-30T06:26:32.323ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
59452025-04-30T06:26:32.335ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
59462025-04-30T06:26:32.339ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85763d6d-58c1-47c8-ab9f-eadf771f1a07", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59472025-04-30T06:26:32.339ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
59482025-04-30T06:26:32.340ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0172ea1c-3be7-4992-bce1-f546d5ca7977", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59492025-04-30T06:26:32.341ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
59502025-04-30T06:26:32.355ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca3a64f4-bf0a-4e73-8b7e-5a4d842b6b31", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59512025-04-30T06:26:32.355ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
59522025-04-30T06:26:32.356ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dbe6491-31df-4187-9852-70b33314ca03", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59532025-04-30T06:26:32.356ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
59542025-04-30T06:26:32.357ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54687770-720c-4d35-b5a3-e4ea6cc067e7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59552025-04-30T06:26:32.357ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
59562025-04-30T06:26:32.359ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbadb5c5-a544-4412-911c-0184d0eff009", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59572025-04-30T06:26:32.359ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
59582025-04-30T06:26:32.359ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d22b82e1-4a77-41f6-8360-1f87e7e6db88", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59592025-04-30T06:26:32.361ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
59602025-04-30T06:26:32.362ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ec84221-eb2c-4bea-aedc-ac469663d650", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59612025-04-30T06:26:32.362ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
59622025-04-30T06:26:32.363ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e58fa97-dfcc-4d50-8992-928c50f626ce", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59632025-04-30T06:26:32.363ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
59642025-04-30T06:26:32.365ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfd6d49f-d793-41fe-9ec1-023978a28968", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59652025-04-30T06:26:32.365ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
59662025-04-30T06:26:32.366ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0b2d4c9-f2f3-49f5-b5b0-257cdf4c0e39", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59672025-04-30T06:26:32.366ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
59682025-04-30T06:26:32.368ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee8ebcbc-cd3e-47c7-b785-7e7bf74f8227", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59692025-04-30T06:26:32.368ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
59702025-04-30T06:26:32.371ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da7bd987-07a8-4862-bf02-f6d4e83782aa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59712025-04-30T06:26:32.371ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
59722025-04-30T06:26:32.371ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f5fc7bb-2dc0-408c-b879-5225fb920769", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59732025-04-30T06:26:32.371ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
59742025-04-30T06:26:32.372ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87cdeb20-852a-4f7e-bd9a-77dac7a00234", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59752025-04-30T06:26:32.373ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
59762025-04-30T06:26:32.373ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f7b3594-eb0c-44e8-b3af-c96d51b525bd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59772025-04-30T06:26:32.373ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
59782025-04-30T06:26:32.374ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "097bf643-5f03-4854-abe1-03f2581f497b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
59792025-04-30T06:26:32.374ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59802025-04-30T06:26:32.374ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59812025-04-30T06:26:32.374ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59822025-04-30T06:26:32.374ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59832025-04-30T06:26:32.375ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59842025-04-30T06:26:32.377ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59852025-04-30T06:26:32.381ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59862025-04-30T06:26:32.381ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59872025-04-30T06:26:32.381ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59882025-04-30T06:26:32.381ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59892025-04-30T06:26:32.381ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59902025-04-30T06:26:32.382ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59912025-04-30T06:26:32.384ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59922025-04-30T06:26:32.390ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59932025-04-30T06:26:32.390ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59942025-04-30T06:26:32.390ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59952025-04-30T06:26:32.391ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59962025-04-30T06:26:32.391ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59972025-04-30T06:26:32.392ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59982025-04-30T06:26:32.392ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59992025-04-30T06:26:32.392ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
60002025-04-30T06:26:32.393ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60012025-04-30T06:26:32.395ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60022025-04-30T06:26:32.401ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60032025-04-30T06:26:32.401ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60042025-04-30T06:26:32.401ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
60052025-04-30T06:26:32.401ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60062025-04-30T06:26:32.466ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
60072025-04-30T06:26:32.467ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
60082025-04-30T06:26:32.468ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:59559 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
60092025-04-30T06:26:32.469ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60102025-04-30T06:26:32.469ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
60112025-04-30T06:26:32.470ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60122025-04-30T06:26:32.470ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:59559 req_id = e8afc566-2383-4ee0-8ed1-82429231652e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
60132025-04-30T06:26:32.471ZINFOtest_disk_expunge (dropshot): request completed latency_us = 697 local_addr = [::1]:44559 method = GET remote_addr = [::1]:59559 req_id = e8afc566-2383-4ee0-8ed1-82429231652e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
60142025-04-30T06:26:32.471ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8afc566-2383-4ee0-8ed1-82429231652e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
60152025-04-30T06:26:32.475ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
60162025-04-30T06:26:32.475ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
60172025-04-30T06:26:32.475ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
60182025-04-30T06:26:32.477ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
60192025-04-30T06:26:32.477ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60202025-04-30T06:26:32.477ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60212025-04-30T06:26:32.478ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60222025-04-30T06:26:32.478ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60232025-04-30T06:26:32.482ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60242025-04-30T06:26:32.487ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
60252025-04-30T06:26:32.487ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60262025-04-30T06:26:32.487ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60272025-04-30T06:26:32.488ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60282025-04-30T06:26:32.518ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
60292025-04-30T06:26:32.521ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
60302025-04-30T06:26:32.522ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
60312025-04-30T06:26:32.522ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 546.512591ms iteration = 16 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
60322025-04-30T06:26:32.523ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 17 reason = Signaled
60332025-04-30T06:26:32.523ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
60342025-04-30T06:26:32.523ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60352025-04-30T06:26:32.523ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60362025-04-30T06:26:32.523ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60372025-04-30T06:26:32.523ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60382025-04-30T06:26:32.527ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60392025-04-30T06:26:32.536ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
60402025-04-30T06:26:32.536ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60412025-04-30T06:26:32.536ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60422025-04-30T06:26:32.536ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60432025-04-30T06:26:32.540ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
60442025-04-30T06:26:32.540ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
60452025-04-30T06:26:32.541ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 90 33 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .3..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62873 req_id = 73a6087f-418a-4736-a378-2f5dea95af8a
60462025-04-30T06:26:32.541ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36915,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62873 req_id = 73a6087f-418a-4736-a378-2f5dea95af8a
60472025-04-30T06:26:32.542ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
60482025-04-30T06:26:32.542ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60492025-04-30T06:26:32.542ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62873 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 73a6087f-418a-4736-a378-2f5dea95af8a
60502025-04-30T06:26:32.542ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3257, tv_nsec: 341800032 } })
60512025-04-30T06:26:32.543ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
60522025-04-30T06:26:32.543ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f3 2f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ./..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58776 req_id = ae9e4f5a-6251-4c35-9349-eb7823fd00b3
60532025-04-30T06:26:32.543ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62255,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58776 req_id = ae9e4f5a-6251-4c35-9349-eb7823fd00b3
60542025-04-30T06:26:32.543ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60552025-04-30T06:26:32.543ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58776 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ae9e4f5a-6251-4c35-9349-eb7823fd00b3
60562025-04-30T06:26:32.611ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
60572025-04-30T06:26:32.611ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60582025-04-30T06:26:32.611ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60592025-04-30T06:26:32.611ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60602025-04-30T06:26:32.611ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
60612025-04-30T06:26:32.612ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60622025-04-30T06:26:32.612ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f7 80 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64174 req_id = 21447aa2-4c94-4e1c-864f-12d489839850
60632025-04-30T06:26:32.613ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63360,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64174 req_id = 21447aa2-4c94-4e1c-864f-12d489839850
60642025-04-30T06:26:32.613ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
60652025-04-30T06:26:32.613ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63360, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64174 req_id = 21447aa2-4c94-4e1c-864f-12d489839850
60662025-04-30T06:26:32.613ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
60672025-04-30T06:26:32.613ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
60682025-04-30T06:26:32.613ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
60692025-04-30T06:26:32.615ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60702025-04-30T06:26:32.622ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60712025-04-30T06:26:32.622ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60722025-04-30T06:26:32.623ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
60732025-04-30T06:26:32.624ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60742025-04-30T06:26:32.627ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8815c5d4-7530-4a01-b9e0-628384f78fd4", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
60752025-04-30T06:26:32.628ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
60762025-04-30T06:26:32.629ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60772025-04-30T06:26:32.629ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
60782025-04-30T06:26:32.630ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60792025-04-30T06:26:32.630ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa50e2fd-f0cf-41e9-855b-92ca770f7568", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
60802025-04-30T06:26:32.630ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
60812025-04-30T06:26:32.634ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a72441d-782d-4de5-a0ad-ce4cbf57bedb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
60822025-04-30T06:26:32.634ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
60832025-04-30T06:26:32.644ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22239ba1-e84f-49ee-8406-5cdbd58195b2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
60842025-04-30T06:26:32.645ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
60852025-04-30T06:26:32.648ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36a4bae6-b6ce-458e-bd07-6e5508d90812", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
60862025-04-30T06:26:32.649ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
60872025-04-30T06:26:32.653ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52390948-48b3-4c71-b487-ee21ccd72e37", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
60882025-04-30T06:26:32.653ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
60892025-04-30T06:26:32.659ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ddb16de-27d0-4dad-9cb8-652d5929f424", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
60902025-04-30T06:26:32.659ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
60912025-04-30T06:26:32.662ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66e477e0-560e-474e-995c-8a90ff22dc08", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
60922025-04-30T06:26:32.662ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
60932025-04-30T06:26:32.664ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "222f88f2-f736-4b44-aaba-4e1426d35aad", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
60942025-04-30T06:26:32.664ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
60952025-04-30T06:26:32.665ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bf65215-d9f8-49c5-9d7b-f01869ce529a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
60962025-04-30T06:26:32.665ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
60972025-04-30T06:26:32.668ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db61657e-c7d9-40db-ad42-86894d748515", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
60982025-04-30T06:26:32.668ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
60992025-04-30T06:26:32.669ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0451bb22-2735-4c72-bad8-7abd5f4e47bf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61002025-04-30T06:26:32.669ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
61012025-04-30T06:26:32.670ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "841db186-0795-490e-b218-49f9a83a1e31", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61022025-04-30T06:26:32.671ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
61032025-04-30T06:26:32.675ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
61042025-04-30T06:26:32.677ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "caed896d-67d6-4045-a5dd-c92499266059", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61052025-04-30T06:26:32.677ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
61062025-04-30T06:26:32.680ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9def8fe-6847-4d64-873e-b3664458d06a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61072025-04-30T06:26:32.680ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
61082025-04-30T06:26:32.682ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6d4cc8e-bb6e-47de-92d2-729045f9d0d4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61092025-04-30T06:26:32.682ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
61102025-04-30T06:26:32.684ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b6bfa0f-975b-49d9-a07b-af6575d2709e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61112025-04-30T06:26:32.684ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
61122025-04-30T06:26:32.686ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64c95636-eebc-4bc2-875c-d6a906626e6d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61132025-04-30T06:26:32.686ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
61142025-04-30T06:26:32.689ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66599eb8-531d-496e-8ace-581a5dc662d7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61152025-04-30T06:26:32.690ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
61162025-04-30T06:26:32.691ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b7e1184-6afa-4ddf-89e0-6d8452912875", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61172025-04-30T06:26:32.691ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
61182025-04-30T06:26:32.692ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f160222a-c11d-4123-b29f-27f8a2d27396", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61192025-04-30T06:26:32.692ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
61202025-04-30T06:26:32.693ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcb20015-1753-488e-a5ba-277e426202cd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61212025-04-30T06:26:32.694ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
61222025-04-30T06:26:32.695ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f5a0531-702e-44ca-88b5-9e09b8023782", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61232025-04-30T06:26:32.695ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
61242025-04-30T06:26:32.697ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de1be385-2017-4aee-ab11-384f7ce6695f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61252025-04-30T06:26:32.697ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
61262025-04-30T06:26:32.701ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1ff70cc-44c7-4b9b-9ebc-d60e3f858516", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61272025-04-30T06:26:32.701ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
61282025-04-30T06:26:32.703ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f527323a-efec-4a3e-a09f-616ba58dcc1a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61292025-04-30T06:26:32.703ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
61302025-04-30T06:26:32.705ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5726c481-4852-44e1-917b-2cbc04e17ff1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61312025-04-30T06:26:32.706ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
61322025-04-30T06:26:32.707ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "191095ad-d3e9-48d1-bbb5-8853f55d5992", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61332025-04-30T06:26:32.708ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
61342025-04-30T06:26:32.710ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a3bcaae-9167-4dd3-912b-c2ff982b6f73", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61352025-04-30T06:26:32.710ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
61362025-04-30T06:26:32.713ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91efd1de-17f5-4d35-9f7d-804a6cd857cd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61372025-04-30T06:26:32.714ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
61382025-04-30T06:26:32.716ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f124831e-d384-4d03-be91-c6f65a1240c9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61392025-04-30T06:26:32.716ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
61402025-04-30T06:26:32.717ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a87fcac6-fcb3-4eb4-8562-37f3a5464f8f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61412025-04-30T06:26:32.718ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
61422025-04-30T06:26:32.719ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5df50ad4-5612-42d8-8031-99d524fe6121", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61432025-04-30T06:26:32.719ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
61442025-04-30T06:26:32.719ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7613cbb2-0f6b-4199-a920-9f33b10c8aa7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61452025-04-30T06:26:32.719ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
61462025-04-30T06:26:32.721ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dc5fe1a-bcca-48c8-ab58-46c538d6b894", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61472025-04-30T06:26:32.722ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
61482025-04-30T06:26:32.724ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2b5b9a3-7112-4046-9658-deda4458562c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61492025-04-30T06:26:32.724ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
61502025-04-30T06:26:32.726ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61512025-04-30T06:26:32.726ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61522025-04-30T06:26:32.726ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61532025-04-30T06:26:32.726ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
61542025-04-30T06:26:32.726ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61552025-04-30T06:26:32.728ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "736657e5-d25f-4b73-b0dc-ec45a75a8241", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61562025-04-30T06:26:32.729ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
61572025-04-30T06:26:32.729ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61582025-04-30T06:26:32.736ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
61592025-04-30T06:26:32.736ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61602025-04-30T06:26:32.736ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
61612025-04-30T06:26:32.736ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61622025-04-30T06:26:32.740ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61632025-04-30T06:26:32.740ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
61642025-04-30T06:26:32.741ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61652025-04-30T06:26:32.741ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "699dbff7-29d3-49f0-b7d6-ca30523fcfb9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61662025-04-30T06:26:32.741ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
61672025-04-30T06:26:32.744ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b83d537a-15a7-4413-afe7-e41e479a33ca", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61682025-04-30T06:26:32.744ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
61692025-04-30T06:26:32.747ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49a9622c-44bf-4539-9e87-8c4d7574311e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61702025-04-30T06:26:32.747ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
61712025-04-30T06:26:32.751ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3e84542-14ae-41ad-8eb4-d6a436a35e50", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61722025-04-30T06:26:32.751ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
61732025-04-30T06:26:32.752ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b07747ef-52b0-4674-8a5d-5e796bcdb763", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61742025-04-30T06:26:32.753ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
61752025-04-30T06:26:32.754ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5aa4e5b3-c147-4d85-93bf-43ec07fbb28b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61762025-04-30T06:26:32.754ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
61772025-04-30T06:26:32.755ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89a62b96-1ec2-4e4f-9d9b-872e9cbbf1c6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61782025-04-30T06:26:32.756ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
61792025-04-30T06:26:32.757ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8d22f9f-9e6a-4679-9f35-09ca6a24917e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
61802025-04-30T06:26:32.757ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61812025-04-30T06:26:32.757ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61822025-04-30T06:26:32.757ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61832025-04-30T06:26:32.757ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61842025-04-30T06:26:32.758ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61852025-04-30T06:26:32.761ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61862025-04-30T06:26:32.772ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61872025-04-30T06:26:32.772ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61882025-04-30T06:26:32.772ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61892025-04-30T06:26:32.773ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61902025-04-30T06:26:32.773ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61912025-04-30T06:26:32.774ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61922025-04-30T06:26:32.778ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61932025-04-30T06:26:32.783ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61942025-04-30T06:26:32.783ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61952025-04-30T06:26:32.783ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61962025-04-30T06:26:32.784ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61972025-04-30T06:26:32.869ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
61982025-04-30T06:26:32.869ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
61992025-04-30T06:26:32.879ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:51162 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
62002025-04-30T06:26:32.884ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
62012025-04-30T06:26:32.885ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:51162 req_id = 7ad4c243-7884-4054-a529-41065b1a8468 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62022025-04-30T06:26:32.886ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
62032025-04-30T06:26:32.889ZINFOtest_disk_expunge (dropshot): request completed latency_us = 3656 local_addr = [::1]:44559 method = GET remote_addr = [::1]:51162 req_id = 7ad4c243-7884-4054-a529-41065b1a8468 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62042025-04-30T06:26:32.890ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
62052025-04-30T06:26:32.890ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = b56e1ce8-d1c7-4849-829e-5db458ab6d43 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
62062025-04-30T06:26:32.891ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = b56e1ce8-d1c7-4849-829e-5db458ab6d43 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
62072025-04-30T06:26:32.891ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = b56e1ce8-d1c7-4849-829e-5db458ab6d43 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
62082025-04-30T06:26:32.892ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = b56e1ce8-d1c7-4849-829e-5db458ab6d43 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
62092025-04-30T06:26:32.892ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ad4c243-7884-4054-a529-41065b1a8468", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
62102025-04-30T06:26:32.895ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62112025-04-30T06:26:32.895ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62122025-04-30T06:26:32.895ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
62132025-04-30T06:26:32.909ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62142025-04-30T06:26:32.909ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62152025-04-30T06:26:32.909ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62162025-04-30T06:26:32.909ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62172025-04-30T06:26:32.909ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62182025-04-30T06:26:32.918ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 28244 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = b56e1ce8-d1c7-4849-829e-5db458ab6d43 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
62192025-04-30T06:26:32.920ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "b56e1ce8-d1c7-4849-829e-5db458ab6d43", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
62202025-04-30T06:26:32.920ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
62212025-04-30T06:26:32.920ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 6025ba89-58a7-4372-ba02-944f9a6583d0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
62222025-04-30T06:26:32.920ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 6025ba89-58a7-4372-ba02-944f9a6583d0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
62232025-04-30T06:26:32.920ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 6025ba89-58a7-4372-ba02-944f9a6583d0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
62242025-04-30T06:26:32.924ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 6025ba89-58a7-4372-ba02-944f9a6583d0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
62252025-04-30T06:26:32.925ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62262025-04-30T06:26:32.932ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62272025-04-30T06:26:32.932ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62282025-04-30T06:26:32.932ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62292025-04-30T06:26:32.932ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62302025-04-30T06:26:32.965ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62312025-04-30T06:26:32.965ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62322025-04-30T06:26:32.965ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62332025-04-30T06:26:32.965ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
62342025-04-30T06:26:32.966ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62352025-04-30T06:26:32.966ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62362025-04-30T06:26:32.966ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62372025-04-30T06:26:32.967ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
62382025-04-30T06:26:32.968ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
62392025-04-30T06:26:32.968ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 48397 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 6025ba89-58a7-4372-ba02-944f9a6583d0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
62402025-04-30T06:26:32.969ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "6025ba89-58a7-4372-ba02-944f9a6583d0", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
62412025-04-30T06:26:32.969ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
62422025-04-30T06:26:32.969ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62432025-04-30T06:26:32.969ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
62442025-04-30T06:26:32.974ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62452025-04-30T06:26:32.982ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62462025-04-30T06:26:32.982ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62472025-04-30T06:26:32.982ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
62482025-04-30T06:26:32.982ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62492025-04-30T06:26:32.983ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
62502025-04-30T06:26:32.983ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
62512025-04-30T06:26:32.990ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
62522025-04-30T06:26:32.990ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62532025-04-30T06:26:33.036ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
62542025-04-30T06:26:33.036ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
62552025-04-30T06:26:33.036ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 513.466759ms iteration = 17 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
62562025-04-30T06:26:33.036ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 18 reason = Signaled
62572025-04-30T06:26:33.036ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62582025-04-30T06:26:33.036ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62592025-04-30T06:26:33.036ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62602025-04-30T06:26:33.036ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62612025-04-30T06:26:33.037ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62622025-04-30T06:26:33.040ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62632025-04-30T06:26:33.040ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
62642025-04-30T06:26:33.042ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62652025-04-30T06:26:33.044ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62662025-04-30T06:26:33.051ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62672025-04-30T06:26:33.053ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62682025-04-30T06:26:33.057ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62692025-04-30T06:26:33.057ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62702025-04-30T06:26:33.057ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62712025-04-30T06:26:33.057ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62722025-04-30T06:26:33.063ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62732025-04-30T06:26:33.064ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62742025-04-30T06:26:33.064ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62752025-04-30T06:26:33.064ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62762025-04-30T06:26:33.064ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
62772025-04-30T06:26:33.064ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
62782025-04-30T06:26:33.067ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e1 65 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .e..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63490 req_id = a5fced26-6622-4d98-b0ab-7651357636d7
62792025-04-30T06:26:33.067ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57701,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63490 req_id = a5fced26-6622-4d98-b0ab-7651357636d7
62802025-04-30T06:26:33.067ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
62812025-04-30T06:26:33.068ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62822025-04-30T06:26:33.068ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63490 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a5fced26-6622-4d98-b0ab-7651357636d7
62832025-04-30T06:26:33.069ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3257, tv_nsec: 868414257 } })
62842025-04-30T06:26:33.069ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
62852025-04-30T06:26:33.070ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
62862025-04-30T06:26:33.073ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c7 f9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54398 req_id = ab52c982-1554-4131-8364-6582e0cd828e
62872025-04-30T06:26:33.073ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51193,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54398 req_id = ab52c982-1554-4131-8364-6582e0cd828e
62882025-04-30T06:26:33.073ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62892025-04-30T06:26:33.073ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54398 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ab52c982-1554-4131-8364-6582e0cd828e
62902025-04-30T06:26:33.130ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
62912025-04-30T06:26:33.142ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f2 61 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .a..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50144 req_id = 8d31757f-3c0d-4e12-9111-94e6e0886c86
62922025-04-30T06:26:33.143ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62049,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50144 req_id = 8d31757f-3c0d-4e12-9111-94e6e0886c86
62932025-04-30T06:26:33.145ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
62942025-04-30T06:26:33.145ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62049, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50144 req_id = 8d31757f-3c0d-4e12-9111-94e6e0886c86
62952025-04-30T06:26:33.146ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
62962025-04-30T06:26:33.146ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
62972025-04-30T06:26:33.146ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
62982025-04-30T06:26:33.152ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc41d0fa-8f94-47cd-8f06-8660fbc014b5", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:32 GMT"} })
62992025-04-30T06:26:33.152ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
63002025-04-30T06:26:33.155ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "47282996-25ed-4f9a-a9dc-6c722acacdf1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63012025-04-30T06:26:33.156ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
63022025-04-30T06:26:33.160ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98a621c3-99df-4e78-a343-bd71c632a776", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63032025-04-30T06:26:33.161ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
63042025-04-30T06:26:33.163ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "979cc5d6-e2d7-48c7-9a67-48576dd8a80f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63052025-04-30T06:26:33.163ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
63062025-04-30T06:26:33.166ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cce8bf48-2a4a-4a10-b657-36e358fbeaed", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63072025-04-30T06:26:33.166ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
63082025-04-30T06:26:33.168ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ca7b0e1-0e01-4ca9-9861-0aceef25e966", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63092025-04-30T06:26:33.168ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
63102025-04-30T06:26:33.172ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "befb0aae-da13-4fce-bf75-f4b7cf05235a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63112025-04-30T06:26:33.173ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
63122025-04-30T06:26:33.173ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
63132025-04-30T06:26:33.175ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a894f1a0-6a3b-496d-896c-f8901fe9697c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63142025-04-30T06:26:33.176ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
63152025-04-30T06:26:33.177ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67e309cf-0c10-4314-8def-09964f30635a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63162025-04-30T06:26:33.177ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
63172025-04-30T06:26:33.178ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1781e6f-e213-4491-88eb-15ebbc19e6e1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63182025-04-30T06:26:33.178ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
63192025-04-30T06:26:33.179ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38756a9d-478b-463d-9391-6b4fca3d92bf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63202025-04-30T06:26:33.179ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
63212025-04-30T06:26:33.180ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89289f19-836c-4831-8413-05e213dbcb03", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63222025-04-30T06:26:33.180ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
63232025-04-30T06:26:33.181ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7c66ff1-d138-4898-97ea-1b684a5e246c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63242025-04-30T06:26:33.181ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
63252025-04-30T06:26:33.183ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bdcbe0a-1ce9-4e9d-94eb-31c51bd6b9d7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63262025-04-30T06:26:33.183ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
63272025-04-30T06:26:33.185ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46794164-b6e4-4968-98ec-2f2c9535233f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63282025-04-30T06:26:33.185ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
63292025-04-30T06:26:33.187ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d149bb19-7d97-4930-a33b-4907c17222cf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63302025-04-30T06:26:33.187ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
63312025-04-30T06:26:33.190ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f129ebb7-835d-4e69-9091-288869dc5a84", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63322025-04-30T06:26:33.190ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
63332025-04-30T06:26:33.192ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "006cba6d-1619-474f-94dd-daeb93791353", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63342025-04-30T06:26:33.192ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
63352025-04-30T06:26:33.194ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1ddfcad-998b-4712-8195-fcc2769d9684", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63362025-04-30T06:26:33.194ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
63372025-04-30T06:26:33.195ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57993e8d-5c0b-4012-94c8-296ff3c1ab73", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63382025-04-30T06:26:33.195ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
63392025-04-30T06:26:33.197ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a87c3108-ed15-4beb-b259-0e8be5e854b3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63402025-04-30T06:26:33.197ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
63412025-04-30T06:26:33.199ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3f72e48-859b-484a-82b4-be7972961918", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63422025-04-30T06:26:33.199ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
63432025-04-30T06:26:33.199ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8c7b414-787e-4d2a-92d3-d4ca8f6074e9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63442025-04-30T06:26:33.200ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
63452025-04-30T06:26:33.202ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2b99888-47c8-4333-93b8-abef9c186d14", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63462025-04-30T06:26:33.202ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
63472025-04-30T06:26:33.204ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdb132a2-1755-4a6c-9b06-957d7fd46c33", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63482025-04-30T06:26:33.205ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
63492025-04-30T06:26:33.207ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9fde8ee-4e8b-49c5-8aa6-cc96442960a6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63502025-04-30T06:26:33.207ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
63512025-04-30T06:26:33.209ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59921bc3-b815-4b38-9264-1cdf3e2090ba", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63522025-04-30T06:26:33.209ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
63532025-04-30T06:26:33.213ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05fd26d9-4bb3-484e-966c-4c5dccce5c10", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63542025-04-30T06:26:33.213ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
63552025-04-30T06:26:33.217ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "47e7d033-52aa-4dc1-b25b-d98d92e11021", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63562025-04-30T06:26:33.217ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
63572025-04-30T06:26:33.219ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b364c37e-194f-4997-930f-e8d22e96dba5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63582025-04-30T06:26:33.219ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
63592025-04-30T06:26:33.220ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c1b8402-566a-401c-b4ff-c37402b29bdd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63602025-04-30T06:26:33.220ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
63612025-04-30T06:26:33.222ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9627fba8-3f1b-46b7-bbf7-53d18f0022c7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63622025-04-30T06:26:33.222ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
63632025-04-30T06:26:33.223ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "576784de-fea3-43a6-87d0-b1257219374e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63642025-04-30T06:26:33.223ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
63652025-04-30T06:26:33.224ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63662025-04-30T06:26:33.224ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63672025-04-30T06:26:33.224ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63682025-04-30T06:26:33.224ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
63692025-04-30T06:26:33.225ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63702025-04-30T06:26:33.226ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25e98e30-ea2a-490c-b8f8-48e19be7c4bf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63712025-04-30T06:26:33.226ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
63722025-04-30T06:26:33.227ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0064c07-b8be-4575-b56b-011dc4d2e4e6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63732025-04-30T06:26:33.227ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
63742025-04-30T06:26:33.230ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63752025-04-30T06:26:33.243ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63762025-04-30T06:26:33.243ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63772025-04-30T06:26:33.243ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
63782025-04-30T06:26:33.244ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63792025-04-30T06:26:33.247ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63802025-04-30T06:26:33.247ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
63812025-04-30T06:26:33.248ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63822025-04-30T06:26:33.248ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a660bd80-ecdc-41e5-bfe9-8c4c20e55e9b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63832025-04-30T06:26:33.248ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
63842025-04-30T06:26:33.252ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0741054c-41c8-4878-a1b0-17850c0c6427", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63852025-04-30T06:26:33.252ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
63862025-04-30T06:26:33.254ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49651999-f8d2-4ede-92e9-8b42b3aac366", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63872025-04-30T06:26:33.255ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
63882025-04-30T06:26:33.258ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a73b70b-4248-4254-9255-5759919198ce", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63892025-04-30T06:26:33.258ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
63902025-04-30T06:26:33.262ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ded2bb6-6d38-4d33-854e-5e9bcb3614fc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63912025-04-30T06:26:33.263ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
63922025-04-30T06:26:33.266ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b96f1cb-efff-42e6-bd7a-f89e2b792408", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63932025-04-30T06:26:33.266ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
63942025-04-30T06:26:33.267ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73e8373e-a46a-455a-9e7f-fcdc7b9ab4e6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63952025-04-30T06:26:33.267ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
63962025-04-30T06:26:33.268ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cf4526c-6924-41f0-889c-2dbeceeb870e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63972025-04-30T06:26:33.275ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
63982025-04-30T06:26:33.276ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34b65c4e-3cdf-4ec8-9008-cf504d2b2fe8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
63992025-04-30T06:26:33.277ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
64002025-04-30T06:26:33.279ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c79fda5-633b-4b2f-b6f4-76895ba42730", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
64012025-04-30T06:26:33.279ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
64022025-04-30T06:26:33.279ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64032025-04-30T06:26:33.279ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64042025-04-30T06:26:33.279ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64052025-04-30T06:26:33.279ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64062025-04-30T06:26:33.281ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64072025-04-30T06:26:33.285ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
64082025-04-30T06:26:33.285ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
64092025-04-30T06:26:33.286ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64102025-04-30T06:26:33.286ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64112025-04-30T06:26:33.286ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64122025-04-30T06:26:33.286ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64132025-04-30T06:26:33.292ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64142025-04-30T06:26:33.297ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
64152025-04-30T06:26:33.297ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64162025-04-30T06:26:33.297ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64172025-04-30T06:26:33.297ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64182025-04-30T06:26:33.298ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
64192025-04-30T06:26:33.338ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
64202025-04-30T06:26:33.338ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
64212025-04-30T06:26:33.339ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:34034 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
64222025-04-30T06:26:33.339ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:34034 req_id = c1443f68-b05b-4f53-b49c-e44a6dfbe0aa sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
64232025-04-30T06:26:33.339ZINFOtest_disk_expunge (dropshot): request completed latency_us = 447 local_addr = [::1]:44559 method = GET remote_addr = [::1]:34034 req_id = c1443f68-b05b-4f53-b49c-e44a6dfbe0aa response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
64242025-04-30T06:26:33.340ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1443f68-b05b-4f53-b49c-e44a6dfbe0aa", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
64252025-04-30T06:26:33.341ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
64262025-04-30T06:26:33.341ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
64272025-04-30T06:26:33.341ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
64282025-04-30T06:26:33.342ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64292025-04-30T06:26:33.342ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64302025-04-30T06:26:33.342ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64312025-04-30T06:26:33.342ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64322025-04-30T06:26:33.343ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64332025-04-30T06:26:33.346ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64342025-04-30T06:26:33.352ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64352025-04-30T06:26:33.352ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64362025-04-30T06:26:33.352ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64372025-04-30T06:26:33.353ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64382025-04-30T06:26:33.353ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64392025-04-30T06:26:33.353ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64402025-04-30T06:26:33.353ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64412025-04-30T06:26:33.353ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
64422025-04-30T06:26:33.354ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64432025-04-30T06:26:33.358ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64442025-04-30T06:26:33.362ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64452025-04-30T06:26:33.362ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64462025-04-30T06:26:33.362ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
64472025-04-30T06:26:33.363ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64482025-04-30T06:26:33.365ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64492025-04-30T06:26:33.365ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
64502025-04-30T06:26:33.366ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64512025-04-30T06:26:33.400ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
64522025-04-30T06:26:33.401ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
64532025-04-30T06:26:33.401ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 364.55871ms iteration = 18 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
64542025-04-30T06:26:33.401ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 19 reason = Signaled
64552025-04-30T06:26:33.401ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64562025-04-30T06:26:33.401ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64572025-04-30T06:26:33.401ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64582025-04-30T06:26:33.401ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64592025-04-30T06:26:33.402ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64602025-04-30T06:26:33.404ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64612025-04-30T06:26:33.408ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64622025-04-30T06:26:33.408ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64632025-04-30T06:26:33.408ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64642025-04-30T06:26:33.408ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64652025-04-30T06:26:33.413ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
64662025-04-30T06:26:33.414ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
64672025-04-30T06:26:33.415ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
64682025-04-30T06:26:33.417ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 56 35 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 V5..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56304 req_id = 4f382c76-7686-48a7-9a0f-d09d53de71de
64692025-04-30T06:26:33.417ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22069,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56304 req_id = 4f382c76-7686-48a7-9a0f-d09d53de71de
64702025-04-30T06:26:33.417ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
64712025-04-30T06:26:33.417ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64722025-04-30T06:26:33.419ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56304 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4f382c76-7686-48a7-9a0f-d09d53de71de
64732025-04-30T06:26:33.419ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3258, tv_nsec: 218874205 } })
64742025-04-30T06:26:33.420ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
64752025-04-30T06:26:33.420ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 15 d6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53776 req_id = 20f55aa3-605f-4d50-a480-3ac4be8e7cce
64762025-04-30T06:26:33.420ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5590,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53776 req_id = 20f55aa3-605f-4d50-a480-3ac4be8e7cce
64772025-04-30T06:26:33.420ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64782025-04-30T06:26:33.420ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53776 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 20f55aa3-605f-4d50-a480-3ac4be8e7cce
64792025-04-30T06:26:33.472ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
64802025-04-30T06:26:33.473ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64812025-04-30T06:26:33.473ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64822025-04-30T06:26:33.473ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64832025-04-30T06:26:33.473ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
64842025-04-30T06:26:33.474ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64852025-04-30T06:26:33.475ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c8 1e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54493 req_id = e85ad545-23f7-47f2-82a9-f15a88658be5
64862025-04-30T06:26:33.475ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51230,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54493 req_id = e85ad545-23f7-47f2-82a9-f15a88658be5
64872025-04-30T06:26:33.475ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
64882025-04-30T06:26:33.475ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51230, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54493 req_id = e85ad545-23f7-47f2-82a9-f15a88658be5
64892025-04-30T06:26:33.475ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
64902025-04-30T06:26:33.475ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
64912025-04-30T06:26:33.476ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
64922025-04-30T06:26:33.477ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64932025-04-30T06:26:33.483ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64942025-04-30T06:26:33.483ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64952025-04-30T06:26:33.483ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
64962025-04-30T06:26:33.484ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64972025-04-30T06:26:33.486ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbf5e7fe-6428-4d93-a896-fe6005f115d0", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
64982025-04-30T06:26:33.486ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
64992025-04-30T06:26:33.488ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65002025-04-30T06:26:33.488ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
65012025-04-30T06:26:33.489ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65022025-04-30T06:26:33.490ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea42b7f1-f49c-43bc-973d-bafcd5055248", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65032025-04-30T06:26:33.490ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
65042025-04-30T06:26:33.495ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48c5a47c-d406-4210-b34f-b14d79a52b75", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65052025-04-30T06:26:33.495ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
65062025-04-30T06:26:33.507ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69f8cea1-76fa-4120-a459-ac9f8b3c5d7f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65072025-04-30T06:26:33.507ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
65082025-04-30T06:26:33.514ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "278e59c5-270d-4cdb-9ec2-b6fba70dc570", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65092025-04-30T06:26:33.515ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
65102025-04-30T06:26:33.523ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
65112025-04-30T06:26:33.523ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2083ab9f-d5bd-4f32-9040-63e55c050837", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65122025-04-30T06:26:33.523ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
65132025-04-30T06:26:33.527ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b5d726c-62f0-404f-8f91-656b78cf4c1c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65142025-04-30T06:26:33.527ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
65152025-04-30T06:26:33.530ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c44e00c6-5863-4aab-9023-5a7e6b8d5930", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65162025-04-30T06:26:33.530ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
65172025-04-30T06:26:33.531ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94d6914c-7c88-472d-ade4-62eeea38f825", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65182025-04-30T06:26:33.532ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
65192025-04-30T06:26:33.533ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c48b07be-d4b1-4338-a3b4-e6bad2b0defe", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65202025-04-30T06:26:33.533ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
65212025-04-30T06:26:33.535ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd777543-e6af-4726-87a9-188c848bc0b3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65222025-04-30T06:26:33.535ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
65232025-04-30T06:26:33.537ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "916e1fe4-93c5-42af-82bd-987e947b2325", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65242025-04-30T06:26:33.537ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
65252025-04-30T06:26:33.539ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bba50594-346d-4c76-bdf7-8b8fda992d32", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65262025-04-30T06:26:33.539ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
65272025-04-30T06:26:33.544ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49d33a0f-2b62-40e3-9e5a-8708d2d6fd13", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65282025-04-30T06:26:33.544ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
65292025-04-30T06:26:33.548ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "24ef7793-2f97-474c-ad90-caa77ea6b9dd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65302025-04-30T06:26:33.548ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
65312025-04-30T06:26:33.552ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4d9a741-ef88-467a-9910-44ffec9c1a98", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65322025-04-30T06:26:33.553ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
65332025-04-30T06:26:33.556ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ebd129f-b67d-4b9f-96f3-91f5c0f59d30", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65342025-04-30T06:26:33.556ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
65352025-04-30T06:26:33.560ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba78d1c4-481a-4c0e-940f-df6f6aad847b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65362025-04-30T06:26:33.562ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
65372025-04-30T06:26:33.564ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b4793c8-1eda-4578-b0dd-56f97595a704", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65382025-04-30T06:26:33.564ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
65392025-04-30T06:26:33.565ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "406676a4-32e6-486b-ae13-805222038656", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65402025-04-30T06:26:33.565ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
65412025-04-30T06:26:33.566ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ae0773e-bcef-49c5-b1f8-3661ab9455e8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65422025-04-30T06:26:33.566ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
65432025-04-30T06:26:33.567ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23a84b11-dd43-43d8-9ef8-dff0b0fde0fc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65442025-04-30T06:26:33.567ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
65452025-04-30T06:26:33.568ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf11af4d-bdff-474d-83ca-c673dffb059c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65462025-04-30T06:26:33.568ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
65472025-04-30T06:26:33.570ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00dffe78-415e-4fe6-980e-75a51e49cbba", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65482025-04-30T06:26:33.570ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
65492025-04-30T06:26:33.572ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb50e4a3-9d1b-4ade-bafe-01030eb2ead4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65502025-04-30T06:26:33.572ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
65512025-04-30T06:26:33.577ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65522025-04-30T06:26:33.577ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65532025-04-30T06:26:33.577ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65542025-04-30T06:26:33.577ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
65552025-04-30T06:26:33.578ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65562025-04-30T06:26:33.579ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8af44b10-78a9-4feb-b6a3-971618bde27f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65572025-04-30T06:26:33.580ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
65582025-04-30T06:26:33.580ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65592025-04-30T06:26:33.589ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65602025-04-30T06:26:33.589ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65612025-04-30T06:26:33.589ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
65622025-04-30T06:26:33.590ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65632025-04-30T06:26:33.593ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65642025-04-30T06:26:33.593ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
65652025-04-30T06:26:33.594ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65662025-04-30T06:26:33.596ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7c38081-9909-4e8c-a1ee-8eaf2a4ed7d0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65672025-04-30T06:26:33.596ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
65682025-04-30T06:26:33.599ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42d16954-ef2b-4433-82da-3e8f5b03305f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65692025-04-30T06:26:33.600ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
65702025-04-30T06:26:33.605ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "592fd1c8-691e-4a7c-8409-3052291ad4e4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65712025-04-30T06:26:33.605ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
65722025-04-30T06:26:33.607ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f76d01a-fb50-4dce-800f-967d7d294594", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65732025-04-30T06:26:33.607ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
65742025-04-30T06:26:33.610ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ef75f07-6bd5-4aef-9118-dbdeb700b2fe", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65752025-04-30T06:26:33.610ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
65762025-04-30T06:26:33.612ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae1e4336-2fe1-479d-9136-713afbfa8e73", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65772025-04-30T06:26:33.612ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
65782025-04-30T06:26:33.614ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5324f4d-547c-43c6-a1e4-11d9bfe21a67", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65792025-04-30T06:26:33.614ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
65802025-04-30T06:26:33.622ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c298aedb-b52b-4efc-9812-7167b7a396a9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65812025-04-30T06:26:33.622ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
65822025-04-30T06:26:33.625ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a63a2eb7-7da2-4223-ba6a-8750a177f0c8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65832025-04-30T06:26:33.625ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
65842025-04-30T06:26:33.629ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fecdf1f0-36f9-4484-bf5f-d9bcef5fb6b0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65852025-04-30T06:26:33.629ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
65862025-04-30T06:26:33.635ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ced50e53-b75f-42b4-809b-8e887f5843e1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65872025-04-30T06:26:33.635ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
65882025-04-30T06:26:33.636ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
65892025-04-30T06:26:33.638ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6e1e92e-a65e-4d8b-822e-874dee1b0a6a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65902025-04-30T06:26:33.638ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
65912025-04-30T06:26:33.643ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "425a5e62-7a77-4608-97f6-eaebe4d0fcf3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65922025-04-30T06:26:33.643ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
65932025-04-30T06:26:33.646ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20793059-445f-4151-b62d-5152900b42ea", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65942025-04-30T06:26:33.646ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
65952025-04-30T06:26:33.652ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4714363-dfea-4753-8e49-6afa1717fc61", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65962025-04-30T06:26:33.652ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
65972025-04-30T06:26:33.654ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1b9d678-a514-4abc-aaa6-fe7a3f499fd3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
65982025-04-30T06:26:33.654ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
65992025-04-30T06:26:33.655ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9775149a-a021-43eb-9ee2-9afc24f209fb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
66002025-04-30T06:26:33.655ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
66012025-04-30T06:26:33.656ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e5f3cf6-9c29-4e19-bce0-fff5412b4a9b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
66022025-04-30T06:26:33.656ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
66032025-04-30T06:26:33.657ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "276974ca-21ca-4d49-8052-cfad81d66d7e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
66042025-04-30T06:26:33.657ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
66052025-04-30T06:26:33.657ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66062025-04-30T06:26:33.657ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66072025-04-30T06:26:33.657ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66082025-04-30T06:26:33.658ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66092025-04-30T06:26:33.661ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66102025-04-30T06:26:33.666ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
66112025-04-30T06:26:33.666ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
66122025-04-30T06:26:33.666ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66132025-04-30T06:26:33.666ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66142025-04-30T06:26:33.666ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66152025-04-30T06:26:33.667ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66162025-04-30T06:26:33.670ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66172025-04-30T06:26:33.678ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
66182025-04-30T06:26:33.678ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66192025-04-30T06:26:33.678ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66202025-04-30T06:26:33.678ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66212025-04-30T06:26:33.754ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
66222025-04-30T06:26:33.754ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
66232025-04-30T06:26:33.755ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66242025-04-30T06:26:33.755ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66252025-04-30T06:26:33.755ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66262025-04-30T06:26:33.755ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
66272025-04-30T06:26:33.757ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66282025-04-30T06:26:33.757ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:48467 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
66292025-04-30T06:26:33.757ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:48467 req_id = b413d2f2-2567-4773-84ab-90266cd05a5d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
66302025-04-30T06:26:33.758ZINFOtest_disk_expunge (dropshot): request completed latency_us = 458 local_addr = [::1]:44559 method = GET remote_addr = [::1]:48467 req_id = b413d2f2-2567-4773-84ab-90266cd05a5d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
66312025-04-30T06:26:33.758ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b413d2f2-2567-4773-84ab-90266cd05a5d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
66322025-04-30T06:26:33.761ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
66332025-04-30T06:26:33.761ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
66342025-04-30T06:26:33.761ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
66352025-04-30T06:26:33.764ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66362025-04-30T06:26:33.764ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66372025-04-30T06:26:33.764ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66382025-04-30T06:26:33.764ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66392025-04-30T06:26:33.766ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66402025-04-30T06:26:33.768ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66412025-04-30T06:26:33.783ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66422025-04-30T06:26:33.783ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66432025-04-30T06:26:33.783ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
66442025-04-30T06:26:33.787ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66452025-04-30T06:26:33.791ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66462025-04-30T06:26:33.799ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66472025-04-30T06:26:33.800ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66482025-04-30T06:26:33.800ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66492025-04-30T06:26:33.801ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66502025-04-30T06:26:33.803ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66512025-04-30T06:26:33.804ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
66522025-04-30T06:26:33.805ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66532025-04-30T06:26:33.840ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
66542025-04-30T06:26:33.841ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
66552025-04-30T06:26:33.841ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 439.858008ms iteration = 19 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
66562025-04-30T06:26:33.841ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 20 reason = Signaled
66572025-04-30T06:26:33.841ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66582025-04-30T06:26:33.841ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66592025-04-30T06:26:33.841ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66602025-04-30T06:26:33.841ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66612025-04-30T06:26:33.842ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66622025-04-30T06:26:33.848ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66632025-04-30T06:26:33.852ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66642025-04-30T06:26:33.852ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66652025-04-30T06:26:33.852ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66662025-04-30T06:26:33.853ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66672025-04-30T06:26:33.853ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
66682025-04-30T06:26:33.857ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
66692025-04-30T06:26:33.857ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
66702025-04-30T06:26:33.858ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 74 b4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 t............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60430 req_id = 6c9d5146-819a-4721-adef-408999b2a7d5
66712025-04-30T06:26:33.858ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29876,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60430 req_id = 6c9d5146-819a-4721-adef-408999b2a7d5
66722025-04-30T06:26:33.858ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
66732025-04-30T06:26:33.859ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66742025-04-30T06:26:33.859ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60430 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6c9d5146-819a-4721-adef-408999b2a7d5
66752025-04-30T06:26:33.859ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3258, tv_nsec: 658798395 } })
66762025-04-30T06:26:33.860ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
66772025-04-30T06:26:33.860ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6f 8f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e o...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58329 req_id = 1844dfc6-226d-48bf-bc64-96fc92bc9cd1
66782025-04-30T06:26:33.860ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28559,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58329 req_id = 1844dfc6-226d-48bf-bc64-96fc92bc9cd1
66792025-04-30T06:26:33.861ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66802025-04-30T06:26:33.861ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58329 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1844dfc6-226d-48bf-bc64-96fc92bc9cd1
66812025-04-30T06:26:33.939ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
66822025-04-30T06:26:33.939ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66832025-04-30T06:26:33.939ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66842025-04-30T06:26:33.939ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66852025-04-30T06:26:33.939ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
66862025-04-30T06:26:33.940ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66872025-04-30T06:26:33.941ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ad 62 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .b..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53620 req_id = b2810091-9292-419f-9089-6f020a4f0f04
66882025-04-30T06:26:33.941ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44386,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53620 req_id = b2810091-9292-419f-9089-6f020a4f0f04
66892025-04-30T06:26:33.941ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
66902025-04-30T06:26:33.941ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44386, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53620 req_id = b2810091-9292-419f-9089-6f020a4f0f04
66912025-04-30T06:26:33.942ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
66922025-04-30T06:26:33.942ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
66932025-04-30T06:26:33.942ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
66942025-04-30T06:26:33.943ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66952025-04-30T06:26:33.949ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66962025-04-30T06:26:33.949ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66972025-04-30T06:26:33.949ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
66982025-04-30T06:26:33.950ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66992025-04-30T06:26:33.951ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2176672b-9f8f-44c2-a730-12bce4927079", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67002025-04-30T06:26:33.951ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
67012025-04-30T06:26:33.953ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67022025-04-30T06:26:33.953ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
67032025-04-30T06:26:33.955ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67042025-04-30T06:26:33.955ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4186d01-cff4-4071-948b-d8c1c25eb992", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67052025-04-30T06:26:33.955ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
67062025-04-30T06:26:33.956ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
67072025-04-30T06:26:33.956ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
67082025-04-30T06:26:33.956ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
67092025-04-30T06:26:33.957ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
67102025-04-30T06:26:33.962ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
67112025-04-30T06:26:33.962ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
67122025-04-30T06:26:33.965ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
67132025-04-30T06:26:33.966ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
67142025-04-30T06:26:33.967ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b283c980-0903-4251-a1d6-948bc54163ad", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67152025-04-30T06:26:33.968ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
67162025-04-30T06:26:33.969ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0456fe1a-83b0-450b-90ba-7e399c1071b9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67172025-04-30T06:26:33.969ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
67182025-04-30T06:26:33.972ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63ccb7ec-fb8d-4533-b025-03bf86bc00ee", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67192025-04-30T06:26:33.972ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
67202025-04-30T06:26:33.975ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb4ab56e-602d-4fba-b732-ea7302dca158", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67212025-04-30T06:26:33.975ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
67222025-04-30T06:26:33.977ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffc6434d-0864-4c1b-9534-094628d32ae6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67232025-04-30T06:26:33.977ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
67242025-04-30T06:26:33.984ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "514145eb-68db-41db-bb07-14ad22a433a9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67252025-04-30T06:26:33.984ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
67262025-04-30T06:26:33.991ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
67272025-04-30T06:26:33.992ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "826f45a9-408b-474d-bc38-336cb891ca63", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67282025-04-30T06:26:33.992ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
67292025-04-30T06:26:33.994ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b677e269-f738-4309-b972-80220ff59b48", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67302025-04-30T06:26:33.994ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
67312025-04-30T06:26:33.996ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "109f370f-8d19-412d-8a39-1b5fa3902b58", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67322025-04-30T06:26:33.996ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
67332025-04-30T06:26:33.999ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74f6d4e8-f2f4-430f-8dbf-214d27ef3b25", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67342025-04-30T06:26:33.999ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
67352025-04-30T06:26:34.001ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cceb6598-cc1a-4b77-be18-d2d91bd1005e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67362025-04-30T06:26:34.002ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
67372025-04-30T06:26:34.008ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30f3ee4a-a86c-4da5-8937-f7a82a361dcb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67382025-04-30T06:26:34.008ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
67392025-04-30T06:26:34.012ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1cbb73e-23ff-4204-b83d-34d3083b072b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67402025-04-30T06:26:34.012ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
67412025-04-30T06:26:34.014ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30e5d7bf-d799-4e1c-b2da-2581c7faa162", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67422025-04-30T06:26:34.014ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
67432025-04-30T06:26:34.037ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0474ce5c-eacf-4d48-962a-e87412380718", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67442025-04-30T06:26:34.037ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
67452025-04-30T06:26:34.049ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
67462025-04-30T06:26:34.049ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67472025-04-30T06:26:34.049ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67482025-04-30T06:26:34.049ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
67492025-04-30T06:26:34.050ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67502025-04-30T06:26:34.051ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67512025-04-30T06:26:34.051ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67522025-04-30T06:26:34.059ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67532025-04-30T06:26:34.059ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67542025-04-30T06:26:34.059ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67552025-04-30T06:26:34.060ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67562025-04-30T06:26:34.060ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
67572025-04-30T06:26:34.060ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67582025-04-30T06:26:34.067ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
67592025-04-30T06:26:34.067ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67602025-04-30T06:26:34.067ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
67612025-04-30T06:26:34.067ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67622025-04-30T06:26:34.071ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67632025-04-30T06:26:34.071ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
67642025-04-30T06:26:34.072ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67652025-04-30T06:26:34.072ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "322466ba-c96b-4333-9ba4-265564f0a564", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67662025-04-30T06:26:34.072ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
67672025-04-30T06:26:34.079ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1191079-915b-4bba-a50d-a4f1dd2603a9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67682025-04-30T06:26:34.080ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
67692025-04-30T06:26:34.082ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90f9d22d-1caa-41d6-8994-6745269fce62", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67702025-04-30T06:26:34.083ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
67712025-04-30T06:26:34.086ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be26a4b9-b88b-40aa-aa35-d0dfc75a1aa3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67722025-04-30T06:26:34.086ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
67732025-04-30T06:26:34.090ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a10b9eff-f574-43f8-ba3d-0075af7a1885", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67742025-04-30T06:26:34.090ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
67752025-04-30T06:26:34.091ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a24e7073-6203-4f67-91dc-fbf713f5b4b8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67762025-04-30T06:26:34.092ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
67772025-04-30T06:26:34.094ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db31d038-f330-4344-a76b-88ff7e16342b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67782025-04-30T06:26:34.094ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
67792025-04-30T06:26:34.097ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "645d8d12-1b13-4ec0-a2c7-0313f7c43988", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67802025-04-30T06:26:34.097ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
67812025-04-30T06:26:34.099ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdc45248-5a2e-4544-9a29-d84303d1e6c5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67822025-04-30T06:26:34.099ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
67832025-04-30T06:26:34.101ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46918b90-75ef-4f3d-836c-f5a0f49e73b1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67842025-04-30T06:26:34.101ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
67852025-04-30T06:26:34.104ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
67862025-04-30T06:26:34.106ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68f18c28-795e-49ad-aa5c-9a4049e51abd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67872025-04-30T06:26:34.106ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
67882025-04-30T06:26:34.108ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a090081f-2cc4-4c06-bf39-22c9f70f0354", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67892025-04-30T06:26:34.108ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
67902025-04-30T06:26:34.111ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c607caf6-a21d-4ca2-a5ec-cd35b12e8199", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67912025-04-30T06:26:34.111ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
67922025-04-30T06:26:34.113ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c2ed408-5bfc-4f45-a010-9bf5557507ae", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67932025-04-30T06:26:34.114ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
67942025-04-30T06:26:34.115ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb99f496-1161-4303-9f39-c82c2533b3d6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67952025-04-30T06:26:34.115ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
67962025-04-30T06:26:34.117ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bdf75c2-f53a-4f32-9a5b-065bd7dd7c9b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67972025-04-30T06:26:34.117ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
67982025-04-30T06:26:34.119ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f1631b9-1943-4f65-b4c7-4fb2b4cbb4a7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
67992025-04-30T06:26:34.119ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
68002025-04-30T06:26:34.120ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "848a323a-4d77-40cd-995e-1225cc8f4d03", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
68012025-04-30T06:26:34.120ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
68022025-04-30T06:26:34.122ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12f55925-a022-4eec-8607-17de1815b6c9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
68032025-04-30T06:26:34.122ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
68042025-04-30T06:26:34.124ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82454485-74ee-491b-a2b2-7560b61b3b15", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
68052025-04-30T06:26:34.124ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
68062025-04-30T06:26:34.126ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "376899d5-237e-4a26-9cb6-55a75e17a4b6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
68072025-04-30T06:26:34.126ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
68082025-04-30T06:26:34.128ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "283e5755-cd8d-4b91-be93-c3efb1890c34", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
68092025-04-30T06:26:34.133ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
68102025-04-30T06:26:34.133ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38c0878b-cee7-4a47-b65a-27e1599b659e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
68112025-04-30T06:26:34.133ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
68122025-04-30T06:26:34.133ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48d5978d-7522-48cc-abb5-02679aa255ad", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
68132025-04-30T06:26:34.133ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
68142025-04-30T06:26:34.134ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1623fae0-b6f2-4044-a23e-bd08526ada61", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
68152025-04-30T06:26:34.134ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
68162025-04-30T06:26:34.136ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76c205f3-eaca-4a0e-b7ad-3fb6dbde3f62", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
68172025-04-30T06:26:34.136ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
68182025-04-30T06:26:34.138ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f6ed498-0979-43a0-8078-dca8baf0cada", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
68192025-04-30T06:26:34.138ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
68202025-04-30T06:26:34.139ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef11cf20-d4a9-48d1-b598-48cb8583a06e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:33 GMT"} })
68212025-04-30T06:26:34.139ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
68222025-04-30T06:26:34.139ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68232025-04-30T06:26:34.139ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68242025-04-30T06:26:34.139ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68252025-04-30T06:26:34.140ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68262025-04-30T06:26:34.145ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68272025-04-30T06:26:34.154ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
68282025-04-30T06:26:34.155ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
68292025-04-30T06:26:34.155ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68302025-04-30T06:26:34.155ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68312025-04-30T06:26:34.155ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68322025-04-30T06:26:34.156ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68332025-04-30T06:26:34.157ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68342025-04-30T06:26:34.158ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68352025-04-30T06:26:34.158ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68362025-04-30T06:26:34.158ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
68372025-04-30T06:26:34.159ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68382025-04-30T06:26:34.161ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68392025-04-30T06:26:34.169ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
68402025-04-30T06:26:34.170ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68412025-04-30T06:26:34.170ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68422025-04-30T06:26:34.171ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68432025-04-30T06:26:34.172ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68442025-04-30T06:26:34.182ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68452025-04-30T06:26:34.182ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68462025-04-30T06:26:34.182ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
68472025-04-30T06:26:34.184ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68482025-04-30T06:26:34.189ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68492025-04-30T06:26:34.189ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
68502025-04-30T06:26:34.190ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68512025-04-30T06:26:34.252ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
68522025-04-30T06:26:34.252ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
68532025-04-30T06:26:34.253ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:56588 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
68542025-04-30T06:26:34.254ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:56588 req_id = 8a7e1abe-6243-499f-b6bf-752869a79cf9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68552025-04-30T06:26:34.254ZINFOtest_disk_expunge (dropshot): request completed latency_us = 700 local_addr = [::1]:44559 method = GET remote_addr = [::1]:56588 req_id = 8a7e1abe-6243-499f-b6bf-752869a79cf9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68562025-04-30T06:26:34.255ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a7e1abe-6243-499f-b6bf-752869a79cf9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
68572025-04-30T06:26:34.257ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68582025-04-30T06:26:34.257ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68592025-04-30T06:26:34.257ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
68602025-04-30T06:26:34.259ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68612025-04-30T06:26:34.263ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68622025-04-30T06:26:34.263ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68632025-04-30T06:26:34.263ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68642025-04-30T06:26:34.267ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68652025-04-30T06:26:34.272ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68662025-04-30T06:26:34.296ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68672025-04-30T06:26:34.300ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68682025-04-30T06:26:34.300ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68692025-04-30T06:26:34.301ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68702025-04-30T06:26:34.337ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
68712025-04-30T06:26:34.341ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
68722025-04-30T06:26:34.341ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
68732025-04-30T06:26:34.341ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 500.755102ms iteration = 20 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
68742025-04-30T06:26:34.342ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 21 reason = Signaled
68752025-04-30T06:26:34.342ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68762025-04-30T06:26:34.342ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68772025-04-30T06:26:34.342ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68782025-04-30T06:26:34.342ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68792025-04-30T06:26:34.342ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68802025-04-30T06:26:34.346ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68812025-04-30T06:26:34.354ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68822025-04-30T06:26:34.354ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68832025-04-30T06:26:34.354ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68842025-04-30T06:26:34.354ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68852025-04-30T06:26:34.361ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
68862025-04-30T06:26:34.361ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
68872025-04-30T06:26:34.363ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 2b a3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 +............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57852 req_id = f2dea1a9-4d88-4e56-8328-1eee66788813
68882025-04-30T06:26:34.363ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11171,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57852 req_id = f2dea1a9-4d88-4e56-8328-1eee66788813
68892025-04-30T06:26:34.364ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
68902025-04-30T06:26:34.364ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68912025-04-30T06:26:34.364ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57852 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f2dea1a9-4d88-4e56-8328-1eee66788813
68922025-04-30T06:26:34.365ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3259, tv_nsec: 164134334 } })
68932025-04-30T06:26:34.365ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
68942025-04-30T06:26:34.366ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a0 85 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55366 req_id = 2f791fa0-d0de-4c11-a10f-c5be43960596
68952025-04-30T06:26:34.366ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41093,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55366 req_id = 2f791fa0-d0de-4c11-a10f-c5be43960596
68962025-04-30T06:26:34.366ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68972025-04-30T06:26:34.366ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55366 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2f791fa0-d0de-4c11-a10f-c5be43960596
68982025-04-30T06:26:34.437ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
68992025-04-30T06:26:34.438ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69002025-04-30T06:26:34.438ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69012025-04-30T06:26:34.438ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69022025-04-30T06:26:34.438ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
69032025-04-30T06:26:34.438ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69042025-04-30T06:26:34.439ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6c 93 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c l............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60878 req_id = cb384c2b-3d09-44a0-8b4f-2410eb7aae14
69052025-04-30T06:26:34.439ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27795,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60878 req_id = cb384c2b-3d09-44a0-8b4f-2410eb7aae14
69062025-04-30T06:26:34.440ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
69072025-04-30T06:26:34.440ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27795, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60878 req_id = cb384c2b-3d09-44a0-8b4f-2410eb7aae14
69082025-04-30T06:26:34.444ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
69092025-04-30T06:26:34.444ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
69102025-04-30T06:26:34.444ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
69112025-04-30T06:26:34.444ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69122025-04-30T06:26:34.459ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69132025-04-30T06:26:34.459ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69142025-04-30T06:26:34.459ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
69152025-04-30T06:26:34.460ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69162025-04-30T06:26:34.477ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69172025-04-30T06:26:34.477ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
69182025-04-30T06:26:34.478ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69192025-04-30T06:26:34.479ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "828b1ed9-51f4-4a60-8749-01c2ace8f414", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69202025-04-30T06:26:34.479ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
69212025-04-30T06:26:34.485ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eec48485-b210-43e3-934d-5e52ed5677f5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69222025-04-30T06:26:34.485ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
69232025-04-30T06:26:34.493ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71166aae-fc6b-4420-9651-c580f01389e1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69242025-04-30T06:26:34.494ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
69252025-04-30T06:26:34.508ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f93fd1f-a33b-4b4b-adaa-4715b30f28fe", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69262025-04-30T06:26:34.509ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
69272025-04-30T06:26:34.513ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d0e1616-9a5e-4bd3-971a-e989517e0973", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69282025-04-30T06:26:34.513ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
69292025-04-30T06:26:34.520ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
69302025-04-30T06:26:34.522ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a5fcf9e-3c78-42fe-a68e-478165422271", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69312025-04-30T06:26:34.526ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
69322025-04-30T06:26:34.531ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9414c3fa-5d78-4415-ae2e-39ac875f9a47", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69332025-04-30T06:26:34.531ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
69342025-04-30T06:26:34.535ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a3bc6da-acbe-4dc3-9a6d-82a02cae52ce", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69352025-04-30T06:26:34.536ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
69362025-04-30T06:26:34.538ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "458370d1-c9e2-4508-b17e-a590ba4c2acd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69372025-04-30T06:26:34.539ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
69382025-04-30T06:26:34.540ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fdcb9c1-6ff7-48eb-bcef-0849d156e0da", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69392025-04-30T06:26:34.540ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
69402025-04-30T06:26:34.552ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cca08ab6-011c-4631-af1d-4af3d8feea44", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69412025-04-30T06:26:34.552ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
69422025-04-30T06:26:34.554ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "022de02b-6de5-4811-a263-81ce3b424299", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69432025-04-30T06:26:34.554ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
69442025-04-30T06:26:34.558ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cbb6f53-c0b5-4839-8b77-7213137cc0cb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69452025-04-30T06:26:34.558ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
69462025-04-30T06:26:34.564ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "477ca224-69fe-469e-a9df-3aeb6536122b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69472025-04-30T06:26:34.565ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
69482025-04-30T06:26:34.569ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "498dd873-60b5-4c98-b86f-8b4159be180c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69492025-04-30T06:26:34.569ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
69502025-04-30T06:26:34.571ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69512025-04-30T06:26:34.571ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69522025-04-30T06:26:34.571ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69532025-04-30T06:26:34.571ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
69542025-04-30T06:26:34.572ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69552025-04-30T06:26:34.576ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69562025-04-30T06:26:34.597ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69572025-04-30T06:26:34.598ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69582025-04-30T06:26:34.598ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
69592025-04-30T06:26:34.601ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69602025-04-30T06:26:34.604ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d62118c-52d3-4a28-82d1-cc2305eef196", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69612025-04-30T06:26:34.604ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
69622025-04-30T06:26:34.604ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69632025-04-30T06:26:34.604ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
69642025-04-30T06:26:34.606ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69652025-04-30T06:26:34.608ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fc6e34f-d78c-4262-9e83-bdac5fca84ec", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69662025-04-30T06:26:34.609ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
69672025-04-30T06:26:34.612ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce082a60-7212-42d9-98da-a46322e8295d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69682025-04-30T06:26:34.612ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
69692025-04-30T06:26:34.616ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a0b0c99-ed72-4254-9fc7-f0ad25413128", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69702025-04-30T06:26:34.616ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
69712025-04-30T06:26:34.617ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e637560b-e64a-44ec-8bc9-43994a934cbc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69722025-04-30T06:26:34.617ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
69732025-04-30T06:26:34.619ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bd7e53e-530c-487c-be4e-d496177ded51", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69742025-04-30T06:26:34.620ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
69752025-04-30T06:26:34.621ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2909293b-0ab0-4738-9566-6a3fbb6cc2b3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69762025-04-30T06:26:34.621ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
69772025-04-30T06:26:34.623ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "525f3be8-46f6-483b-a59c-a78da8f9dee2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69782025-04-30T06:26:34.623ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
69792025-04-30T06:26:34.626ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b2e3d71-6564-4059-a442-0b2a6473a725", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69802025-04-30T06:26:34.626ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
69812025-04-30T06:26:34.629ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf0058a2-5169-4aa9-ab85-41f8d2bf879e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69822025-04-30T06:26:34.629ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
69832025-04-30T06:26:34.638ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0d06e01-1268-4ac0-8055-380a9ca0e740", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69842025-04-30T06:26:34.638ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
69852025-04-30T06:26:34.644ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
69862025-04-30T06:26:34.647ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "357b07f4-6e19-4683-be94-b3abc7bab6ff", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69872025-04-30T06:26:34.647ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
69882025-04-30T06:26:34.652ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b64390e2-5c6d-488c-9958-69ec1750e6d8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69892025-04-30T06:26:34.652ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
69902025-04-30T06:26:34.662ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6ec18b9-d115-4a2c-a624-a58e2b68db99", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69912025-04-30T06:26:34.662ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
69922025-04-30T06:26:34.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88ca775b-b7bf-477a-939c-a6ece3f16144", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69932025-04-30T06:26:34.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
69942025-04-30T06:26:34.669ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fa03139-5f4b-4023-83a3-249f322a7b80", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69952025-04-30T06:26:34.669ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
69962025-04-30T06:26:34.671ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aed799f6-73b2-4914-afb4-56406db8b992", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69972025-04-30T06:26:34.672ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
69982025-04-30T06:26:34.674ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "154c6666-db6d-420b-acce-ecc2200c162d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
69992025-04-30T06:26:34.674ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
70002025-04-30T06:26:34.676ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6e5f62b-cdec-47b6-9f41-9af383b965c0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70012025-04-30T06:26:34.677ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
70022025-04-30T06:26:34.680ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "558ac4ea-8b43-42d6-9199-bf345e7756d1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70032025-04-30T06:26:34.680ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
70042025-04-30T06:26:34.686ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19270b1f-cfb6-4331-a92b-c3778a913fdc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70052025-04-30T06:26:34.686ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
70062025-04-30T06:26:34.689ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1793631-9363-4287-ac2c-1c1c4573e189", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70072025-04-30T06:26:34.689ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
70082025-04-30T06:26:34.692ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba5fc4f2-665a-451e-81ba-b37e40314065", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70092025-04-30T06:26:34.693ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
70102025-04-30T06:26:34.694ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70112025-04-30T06:26:34.695ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70122025-04-30T06:26:34.695ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70132025-04-30T06:26:34.695ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
70142025-04-30T06:26:34.696ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70152025-04-30T06:26:34.697ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "460e9ebf-2f38-4450-9dca-36602d51fdb9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70162025-04-30T06:26:34.697ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
70172025-04-30T06:26:34.700ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70182025-04-30T06:26:34.711ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70192025-04-30T06:26:34.711ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70202025-04-30T06:26:34.711ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
70212025-04-30T06:26:34.712ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70222025-04-30T06:26:34.714ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70232025-04-30T06:26:34.714ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
70242025-04-30T06:26:34.715ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70252025-04-30T06:26:34.719ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4642fb47-bdce-4220-9e20-025d61c591cc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70262025-04-30T06:26:34.719ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
70272025-04-30T06:26:34.725ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fc9df45-6f18-4a29-a2de-b598e7c9507b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70282025-04-30T06:26:34.725ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
70292025-04-30T06:26:34.726ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad27eeda-67ec-4d5d-a45a-57c403434e45", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70302025-04-30T06:26:34.727ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
70312025-04-30T06:26:34.732ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b37050b-1f7f-4e94-acf9-5b03d9a11d19", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70322025-04-30T06:26:34.733ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
70332025-04-30T06:26:34.736ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34cd471d-056a-4caa-9405-474fe927edbf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70342025-04-30T06:26:34.736ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
70352025-04-30T06:26:34.740ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "062f6fef-dcbd-4d6b-b5f5-51225ca8806c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70362025-04-30T06:26:34.740ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
70372025-04-30T06:26:34.740ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70382025-04-30T06:26:34.740ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70392025-04-30T06:26:34.740ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70402025-04-30T06:26:34.740ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70412025-04-30T06:26:34.743ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70422025-04-30T06:26:34.748ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
70432025-04-30T06:26:34.748ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
70442025-04-30T06:26:34.748ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70452025-04-30T06:26:34.748ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70462025-04-30T06:26:34.748ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70472025-04-30T06:26:34.749ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70482025-04-30T06:26:34.753ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70492025-04-30T06:26:34.758ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
70502025-04-30T06:26:34.758ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70512025-04-30T06:26:34.758ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70522025-04-30T06:26:34.759ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70532025-04-30T06:26:34.759ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
70542025-04-30T06:26:34.822ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
70552025-04-30T06:26:34.823ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
70562025-04-30T06:26:34.824ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70572025-04-30T06:26:34.824ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70582025-04-30T06:26:34.824ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70592025-04-30T06:26:34.824ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
70602025-04-30T06:26:34.825ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70612025-04-30T06:26:34.825ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:41729 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
70622025-04-30T06:26:34.826ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:41729 req_id = 218d8894-1d88-4492-a830-51e4f5ad44d6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70632025-04-30T06:26:34.827ZINFOtest_disk_expunge (dropshot): request completed latency_us = 441 local_addr = [::1]:44559 method = GET remote_addr = [::1]:41729 req_id = 218d8894-1d88-4492-a830-51e4f5ad44d6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70642025-04-30T06:26:34.828ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "218d8894-1d88-4492-a830-51e4f5ad44d6", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70652025-04-30T06:26:34.835ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70662025-04-30T06:26:34.835ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70672025-04-30T06:26:34.835ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
70682025-04-30T06:26:34.839ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70692025-04-30T06:26:34.840ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70702025-04-30T06:26:34.840ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70712025-04-30T06:26:34.840ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70722025-04-30T06:26:34.841ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70732025-04-30T06:26:34.841ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70742025-04-30T06:26:34.854ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70752025-04-30T06:26:34.854ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70762025-04-30T06:26:34.855ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
70772025-04-30T06:26:34.857ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70782025-04-30T06:26:34.860ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70792025-04-30T06:26:34.866ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70802025-04-30T06:26:34.866ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70812025-04-30T06:26:34.866ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70822025-04-30T06:26:34.868ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70832025-04-30T06:26:34.868ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70842025-04-30T06:26:34.868ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
70852025-04-30T06:26:34.869ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70862025-04-30T06:26:34.875ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
70872025-04-30T06:26:34.876ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70882025-04-30T06:26:34.877ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 942a6752-47ff-4efe-8018-9d95bcfcbb6a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70892025-04-30T06:26:34.877ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 942a6752-47ff-4efe-8018-9d95bcfcbb6a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70902025-04-30T06:26:34.877ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 942a6752-47ff-4efe-8018-9d95bcfcbb6a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70912025-04-30T06:26:34.878ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 942a6752-47ff-4efe-8018-9d95bcfcbb6a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70922025-04-30T06:26:34.882ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 5729 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 942a6752-47ff-4efe-8018-9d95bcfcbb6a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70932025-04-30T06:26:34.883ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "942a6752-47ff-4efe-8018-9d95bcfcbb6a", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
70942025-04-30T06:26:34.884ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70952025-04-30T06:26:34.884ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 674b3c07-8b31-46d0-b2fc-d0c4dd5eb897 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70962025-04-30T06:26:34.885ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 674b3c07-8b31-46d0-b2fc-d0c4dd5eb897 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70972025-04-30T06:26:34.885ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 674b3c07-8b31-46d0-b2fc-d0c4dd5eb897 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70982025-04-30T06:26:34.886ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 674b3c07-8b31-46d0-b2fc-d0c4dd5eb897 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70992025-04-30T06:26:34.892ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 8104 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 674b3c07-8b31-46d0-b2fc-d0c4dd5eb897 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
71002025-04-30T06:26:34.894ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "674b3c07-8b31-46d0-b2fc-d0c4dd5eb897", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71012025-04-30T06:26:34.894ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
71022025-04-30T06:26:34.894ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71032025-04-30T06:26:34.894ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
71042025-04-30T06:26:34.904ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
71052025-04-30T06:26:34.904ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
71062025-04-30T06:26:34.904ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 562.733759ms iteration = 21 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
71072025-04-30T06:26:34.904ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 22 reason = Signaled
71082025-04-30T06:26:34.904ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
71092025-04-30T06:26:34.904ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71102025-04-30T06:26:34.904ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71112025-04-30T06:26:34.904ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71122025-04-30T06:26:34.906ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71132025-04-30T06:26:34.910ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
71142025-04-30T06:26:34.911ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71152025-04-30T06:26:34.916ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
71162025-04-30T06:26:34.916ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
71172025-04-30T06:26:34.916ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
71182025-04-30T06:26:34.916ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
71192025-04-30T06:26:34.919ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
71202025-04-30T06:26:34.919ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
71212025-04-30T06:26:34.920ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 38 19 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 8............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65047 req_id = 41ad35fb-0141-409b-b2c7-ddab70ddc5ae
71222025-04-30T06:26:34.920ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14361,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65047 req_id = 41ad35fb-0141-409b-b2c7-ddab70ddc5ae
71232025-04-30T06:26:34.920ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
71242025-04-30T06:26:34.920ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
71252025-04-30T06:26:34.920ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65047 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 41ad35fb-0141-409b-b2c7-ddab70ddc5ae
71262025-04-30T06:26:34.921ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3259, tv_nsec: 720452923 } })
71272025-04-30T06:26:34.921ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
71282025-04-30T06:26:34.921ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f1 09 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56724 req_id = 3d2b8935-5afd-47c0-9a77-a2a90d16de47
71292025-04-30T06:26:34.922ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61705,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56724 req_id = 3d2b8935-5afd-47c0-9a77-a2a90d16de47
71302025-04-30T06:26:34.922ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
71312025-04-30T06:26:34.922ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56724 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3d2b8935-5afd-47c0-9a77-a2a90d16de47
71322025-04-30T06:26:34.987ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
71332025-04-30T06:26:34.989ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71342025-04-30T06:26:34.989ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71352025-04-30T06:26:34.989ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71362025-04-30T06:26:34.989ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
71372025-04-30T06:26:34.990ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71382025-04-30T06:26:34.990ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71392025-04-30T06:26:34.990ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71402025-04-30T06:26:34.993ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
71412025-04-30T06:26:34.995ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
71422025-04-30T06:26:34.997ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 22 50 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c "P..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62223 req_id = 0c32a3c0-7a84-4b4e-83ad-d4b6e23e6ada
71432025-04-30T06:26:34.997ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8784,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62223 req_id = 0c32a3c0-7a84-4b4e-83ad-d4b6e23e6ada
71442025-04-30T06:26:34.997ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
71452025-04-30T06:26:34.997ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8784, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62223 req_id = 0c32a3c0-7a84-4b4e-83ad-d4b6e23e6ada
71462025-04-30T06:26:34.998ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
71472025-04-30T06:26:34.998ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
71482025-04-30T06:26:34.998ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
71492025-04-30T06:26:34.998ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71502025-04-30T06:26:35.009ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71512025-04-30T06:26:35.009ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71522025-04-30T06:26:35.009ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
71532025-04-30T06:26:35.010ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71542025-04-30T06:26:35.011ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
71552025-04-30T06:26:35.011ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
71562025-04-30T06:26:35.014ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71572025-04-30T06:26:35.014ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
71582025-04-30T06:26:35.015ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71592025-04-30T06:26:35.016ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "95ca32a3-0b74-452f-99fe-4f8cff3722b9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71602025-04-30T06:26:35.016ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
71612025-04-30T06:26:35.023ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
71622025-04-30T06:26:35.023ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71632025-04-30T06:26:35.048ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71642025-04-30T06:26:35.048ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c7ba515-3d74-40e6-a627-997631bf2904", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71652025-04-30T06:26:35.049ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
71662025-04-30T06:26:35.049ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71672025-04-30T06:26:35.068ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71682025-04-30T06:26:35.069ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71692025-04-30T06:26:35.070ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71702025-04-30T06:26:35.070ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71712025-04-30T06:26:35.090ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f192b34-2da2-4527-acb6-d5a4d7a1974c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71722025-04-30T06:26:35.090ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
71732025-04-30T06:26:35.091ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
71742025-04-30T06:26:35.093ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bac6879b-4e61-4cc2-81e5-579c092b8273", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71752025-04-30T06:26:35.094ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
71762025-04-30T06:26:35.097ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81b21b59-771a-456f-9688-7aae1715e317", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71772025-04-30T06:26:35.097ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
71782025-04-30T06:26:35.101ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
71792025-04-30T06:26:35.102ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "985cd737-3ed8-4537-b36e-3276dcb01b0a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71802025-04-30T06:26:35.102ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
71812025-04-30T06:26:35.105ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f0883ac-6ab0-4c83-b25a-05c978732810", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71822025-04-30T06:26:35.105ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
71832025-04-30T06:26:35.107ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6e0abee-1296-4d75-bc84-c5573c80efd6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71842025-04-30T06:26:35.107ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
71852025-04-30T06:26:35.108ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cf5fbfb-1fd3-4fee-96c7-5c6953a169ef", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71862025-04-30T06:26:35.108ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
71872025-04-30T06:26:35.109ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e051ff3-8fc7-4b62-ad59-376f0356ca61", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71882025-04-30T06:26:35.109ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
71892025-04-30T06:26:35.110ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7563f73e-5586-44b4-b837-acea4172aaba", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71902025-04-30T06:26:35.110ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
71912025-04-30T06:26:35.111ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9de87d8-5098-410f-99ac-6f6499741510", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71922025-04-30T06:26:35.111ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
71932025-04-30T06:26:35.113ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb310e2e-aae7-43e1-a3b7-d91b4e0d92ba", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71942025-04-30T06:26:35.113ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
71952025-04-30T06:26:35.115ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ab22e46-a413-445e-9363-952d6a414902", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71962025-04-30T06:26:35.116ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
71972025-04-30T06:26:35.118ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a60b52b5-03e6-4974-9a78-ff9f92cafa35", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
71982025-04-30T06:26:35.119ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
71992025-04-30T06:26:35.122ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "643c473c-22fb-4558-b731-f811384d8b0e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72002025-04-30T06:26:35.123ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
72012025-04-30T06:26:35.126ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43176bc8-e81b-4d1a-9201-ac2f34165ad5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72022025-04-30T06:26:35.126ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
72032025-04-30T06:26:35.128ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54950196-6e85-4520-8544-8ffac74609e2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72042025-04-30T06:26:35.128ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
72052025-04-30T06:26:35.131ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef839447-b444-4333-a7e3-0a19c3b9e065", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72062025-04-30T06:26:35.131ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
72072025-04-30T06:26:35.132ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "730e087c-3c6e-4cc4-909c-b93024015bef", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72082025-04-30T06:26:35.132ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
72092025-04-30T06:26:35.133ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "849b7962-0ca7-4cb0-a0ef-6b7f60891544", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72102025-04-30T06:26:35.134ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
72112025-04-30T06:26:35.137ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cce57972-29aa-425d-9459-c718ded2a59f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72122025-04-30T06:26:35.138ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
72132025-04-30T06:26:35.147ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02600a57-5a2b-4c8e-957d-913dd48b2340", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72142025-04-30T06:26:35.147ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
72152025-04-30T06:26:35.149ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "07c8464d-ab67-45e0-9137-4fd27878d8b7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72162025-04-30T06:26:35.149ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
72172025-04-30T06:26:35.151ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8de541a4-4aea-4399-beeb-3bd0192ebdc5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72182025-04-30T06:26:35.151ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
72192025-04-30T06:26:35.153ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72202025-04-30T06:26:35.153ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72212025-04-30T06:26:35.153ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72222025-04-30T06:26:35.153ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
72232025-04-30T06:26:35.155ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72242025-04-30T06:26:35.156ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f99246ea-ea4f-45de-9367-db48e62fe320", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72252025-04-30T06:26:35.156ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
72262025-04-30T06:26:35.159ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72272025-04-30T06:26:35.165ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72282025-04-30T06:26:35.165ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72292025-04-30T06:26:35.165ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
72302025-04-30T06:26:35.166ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72312025-04-30T06:26:35.170ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72322025-04-30T06:26:35.170ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
72332025-04-30T06:26:35.172ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72342025-04-30T06:26:35.173ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fe28ecd-a0f4-441d-a956-2f718064cc59", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72352025-04-30T06:26:35.174ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
72362025-04-30T06:26:35.176ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4e34826-9f42-4dd3-93a6-314e1a400d58", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72372025-04-30T06:26:35.176ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
72382025-04-30T06:26:35.180ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e1fbce3-a173-48d2-83a9-2dce31eca6aa", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72392025-04-30T06:26:35.180ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
72402025-04-30T06:26:35.183ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a6e661b-bfc4-4858-ab76-c4a38f773dae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72412025-04-30T06:26:35.183ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
72422025-04-30T06:26:35.184ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b726b0b-527b-441e-9d71-0d07e5ab6a38", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72432025-04-30T06:26:35.185ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
72442025-04-30T06:26:35.186ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "429cd13e-a942-4dac-9c2a-35b7fdb254e8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72452025-04-30T06:26:35.186ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
72462025-04-30T06:26:35.187ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa8b6378-6dc2-457e-840c-fda31cf94c04", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72472025-04-30T06:26:35.188ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
72482025-04-30T06:26:35.191ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39745637-16a3-4207-ab9f-b467332a5219", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72492025-04-30T06:26:35.191ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
72502025-04-30T06:26:35.192ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "02dac263-50f5-4aae-9da4-3421c60f9354", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72512025-04-30T06:26:35.193ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
72522025-04-30T06:26:35.195ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f19cd27-8954-4f51-ac9d-9814070240e3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72532025-04-30T06:26:35.196ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
72542025-04-30T06:26:35.199ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f5242d9-4213-4452-bf74-c760719b50fd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72552025-04-30T06:26:35.200ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
72562025-04-30T06:26:35.202ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06bba8b4-dfc4-4a5f-8798-fbe1dd8ed474", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72572025-04-30T06:26:35.202ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
72582025-04-30T06:26:35.205ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
72592025-04-30T06:26:35.208ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45065d2a-95b7-4008-8260-3d914d1a7469", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72602025-04-30T06:26:35.208ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
72612025-04-30T06:26:35.211ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "944e272a-f106-401a-b15f-c986cec24853", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72622025-04-30T06:26:35.211ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
72632025-04-30T06:26:35.213ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a56947ce-4631-4567-b8c3-35c29ef9755b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72642025-04-30T06:26:35.213ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
72652025-04-30T06:26:35.215ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc3f563a-11fa-43a5-8d8f-7b95ac3256a1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72662025-04-30T06:26:35.215ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
72672025-04-30T06:26:35.218ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17d3405a-4e22-41af-bd94-82d9ef177dff", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72682025-04-30T06:26:35.218ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
72692025-04-30T06:26:35.218ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cde0963-5d43-4bb8-a41a-8b9e696e3ae4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72702025-04-30T06:26:35.218ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
72712025-04-30T06:26:35.219ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2368498e-6e02-402e-a351-5f6cba64f279", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:34 GMT"} })
72722025-04-30T06:26:35.220ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72732025-04-30T06:26:35.220ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72742025-04-30T06:26:35.220ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72752025-04-30T06:26:35.220ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72762025-04-30T06:26:35.221ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72772025-04-30T06:26:35.224ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72782025-04-30T06:26:35.230ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72792025-04-30T06:26:35.230ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72802025-04-30T06:26:35.230ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72812025-04-30T06:26:35.230ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72822025-04-30T06:26:35.230ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72832025-04-30T06:26:35.231ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72842025-04-30T06:26:35.233ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72852025-04-30T06:26:35.237ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72862025-04-30T06:26:35.237ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72872025-04-30T06:26:35.237ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72882025-04-30T06:26:35.237ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72892025-04-30T06:26:35.319ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
72902025-04-30T06:26:35.319ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
72912025-04-30T06:26:35.319ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72922025-04-30T06:26:35.319ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72932025-04-30T06:26:35.319ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72942025-04-30T06:26:35.319ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
72952025-04-30T06:26:35.324ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72962025-04-30T06:26:35.324ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:62585 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
72972025-04-30T06:26:35.328ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:62585 req_id = 68d1800b-b060-4b56-9d09-064792dfefc1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
72982025-04-30T06:26:35.330ZINFOtest_disk_expunge (dropshot): request completed latency_us = 2607 local_addr = [::1]:44559 method = GET remote_addr = [::1]:62585 req_id = 68d1800b-b060-4b56-9d09-064792dfefc1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
72992025-04-30T06:26:35.330ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73002025-04-30T06:26:35.352ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73012025-04-30T06:26:35.352ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73022025-04-30T06:26:35.352ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
73032025-04-30T06:26:35.352ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73042025-04-30T06:26:35.352ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "68d1800b-b060-4b56-9d09-064792dfefc1", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73052025-04-30T06:26:35.355ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
73062025-04-30T06:26:35.355ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
73072025-04-30T06:26:35.355ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
73082025-04-30T06:26:35.364ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
73092025-04-30T06:26:35.365ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73102025-04-30T06:26:35.365ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73112025-04-30T06:26:35.365ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73122025-04-30T06:26:35.366ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73132025-04-30T06:26:35.371ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73142025-04-30T06:26:35.371ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
73152025-04-30T06:26:35.372ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73162025-04-30T06:26:35.382ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73172025-04-30T06:26:35.396ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
73182025-04-30T06:26:35.397ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73192025-04-30T06:26:35.397ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73202025-04-30T06:26:35.398ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73212025-04-30T06:26:35.517ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
73222025-04-30T06:26:35.519ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
73232025-04-30T06:26:35.519ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
73242025-04-30T06:26:35.519ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 615.194125ms iteration = 22 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
73252025-04-30T06:26:35.520ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 23 reason = Signaled
73262025-04-30T06:26:35.520ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
73272025-04-30T06:26:35.520ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73282025-04-30T06:26:35.521ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73292025-04-30T06:26:35.521ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73302025-04-30T06:26:35.522ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73312025-04-30T06:26:35.526ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73322025-04-30T06:26:35.538ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
73332025-04-30T06:26:35.538ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73342025-04-30T06:26:35.538ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73352025-04-30T06:26:35.539ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73362025-04-30T06:26:35.555ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
73372025-04-30T06:26:35.555ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
73382025-04-30T06:26:35.556ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fd 46 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .F..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64361 req_id = cf56141f-5373-4d5f-a874-887b5535f403
73392025-04-30T06:26:35.556ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64838,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64361 req_id = cf56141f-5373-4d5f-a874-887b5535f403
73402025-04-30T06:26:35.557ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
73412025-04-30T06:26:35.557ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
73422025-04-30T06:26:35.557ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64361 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cf56141f-5373-4d5f-a874-887b5535f403
73432025-04-30T06:26:35.557ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3260, tv_nsec: 356693501 } })
73442025-04-30T06:26:35.557ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
73452025-04-30T06:26:35.558ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 89 bc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63431 req_id = a3c6044a-d106-4ec5-b746-b9b9b1bc04a4
73462025-04-30T06:26:35.558ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35260,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63431 req_id = a3c6044a-d106-4ec5-b746-b9b9b1bc04a4
73472025-04-30T06:26:35.558ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
73482025-04-30T06:26:35.558ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63431 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a3c6044a-d106-4ec5-b746-b9b9b1bc04a4
73492025-04-30T06:26:35.654ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
73502025-04-30T06:26:35.667ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73512025-04-30T06:26:35.667ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73522025-04-30T06:26:35.667ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73532025-04-30T06:26:35.667ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
73542025-04-30T06:26:35.668ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73552025-04-30T06:26:35.668ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3a d0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c :............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58632 req_id = 34dd1b9f-fad7-4688-acd2-9fb3ca815882
73562025-04-30T06:26:35.669ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15056,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58632 req_id = 34dd1b9f-fad7-4688-acd2-9fb3ca815882
73572025-04-30T06:26:35.669ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
73582025-04-30T06:26:35.669ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15056, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58632 req_id = 34dd1b9f-fad7-4688-acd2-9fb3ca815882
73592025-04-30T06:26:35.669ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
73602025-04-30T06:26:35.670ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
73612025-04-30T06:26:35.670ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
73622025-04-30T06:26:35.671ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73632025-04-30T06:26:35.677ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73642025-04-30T06:26:35.677ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73652025-04-30T06:26:35.677ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
73662025-04-30T06:26:35.678ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73672025-04-30T06:26:35.680ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8675c71-e7bc-4cd8-9bc1-f7678c906c47", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73682025-04-30T06:26:35.681ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
73692025-04-30T06:26:35.683ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73702025-04-30T06:26:35.683ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
73712025-04-30T06:26:35.685ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73722025-04-30T06:26:35.685ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62ceb08c-0dd9-4907-b9c7-d24da7f5d91e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73732025-04-30T06:26:35.686ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
73742025-04-30T06:26:35.692ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcc6b121-fc76-44b6-9ed2-031898eb5731", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73752025-04-30T06:26:35.692ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
73762025-04-30T06:26:35.699ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b398e27b-11e4-4cd5-a25e-bd522be66fc6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73772025-04-30T06:26:35.700ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
73782025-04-30T06:26:35.704ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "870638bd-c0cf-4d7e-8469-ab066d887c57", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73792025-04-30T06:26:35.705ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
73802025-04-30T06:26:35.714ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "beea385c-32a8-4ced-b023-657e241629ec", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73812025-04-30T06:26:35.721ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
73822025-04-30T06:26:35.722ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
73832025-04-30T06:26:35.729ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "af62c10a-007f-4855-8218-163b359a3e67", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73842025-04-30T06:26:35.729ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
73852025-04-30T06:26:35.731ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "913fe765-a99c-4229-8a67-b05b22059782", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73862025-04-30T06:26:35.732ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
73872025-04-30T06:26:35.733ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5116aa13-b94c-4290-8d90-23edaddfe9b9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73882025-04-30T06:26:35.733ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
73892025-04-30T06:26:35.735ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d384684c-d968-4e2b-ab01-db791e5516ea", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73902025-04-30T06:26:35.735ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
73912025-04-30T06:26:35.736ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d1caf19-0554-4047-84c6-9c541cc2edad", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73922025-04-30T06:26:35.736ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
73932025-04-30T06:26:35.738ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4c0aba1-82b2-4666-a47b-5c89685c588b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73942025-04-30T06:26:35.738ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
73952025-04-30T06:26:35.740ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee48d168-5d28-4b1e-b3d3-d56f6bf346d0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73962025-04-30T06:26:35.740ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
73972025-04-30T06:26:35.743ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "450b7e13-73d8-42f0-aad3-c9b0dbdc4ed8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
73982025-04-30T06:26:35.743ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
73992025-04-30T06:26:35.755ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "29b84e01-2205-47ee-ba1b-0b86636650d0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74002025-04-30T06:26:35.755ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
74012025-04-30T06:26:35.763ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f66f7d5-a7c9-4d6a-9f49-f198cde1633a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74022025-04-30T06:26:35.763ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
74032025-04-30T06:26:35.765ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8cef5ae-dfa2-4823-b6b3-515e7a0a888f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74042025-04-30T06:26:35.765ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
74052025-04-30T06:26:35.773ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74062025-04-30T06:26:35.773ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74072025-04-30T06:26:35.773ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74082025-04-30T06:26:35.773ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
74092025-04-30T06:26:35.774ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74102025-04-30T06:26:35.776ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e6ecb18-5213-40d8-b970-368afe8421aa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74112025-04-30T06:26:35.777ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
74122025-04-30T06:26:35.779ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74132025-04-30T06:26:35.797ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74142025-04-30T06:26:35.797ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74152025-04-30T06:26:35.798ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
74162025-04-30T06:26:35.799ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74172025-04-30T06:26:35.801ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "557b8366-d81a-42ea-a5eb-adc089f8cc4c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74182025-04-30T06:26:35.801ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
74192025-04-30T06:26:35.802ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74202025-04-30T06:26:35.802ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
74212025-04-30T06:26:35.805ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74222025-04-30T06:26:35.807ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7326f4f2-5e69-4ae9-9659-a87ea53904b0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74232025-04-30T06:26:35.808ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
74242025-04-30T06:26:35.809ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de07edf7-6415-48cf-87d6-8f45ff0fec40", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74252025-04-30T06:26:35.810ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
74262025-04-30T06:26:35.811ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7918e3f0-6c80-4bb7-adcf-4fa57013a987", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74272025-04-30T06:26:35.812ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
74282025-04-30T06:26:35.813ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b45c8bec-2a96-475e-9f11-009410dab298", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74292025-04-30T06:26:35.813ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
74302025-04-30T06:26:35.815ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82e0ae17-0525-4b31-af2d-d32ce08440e2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74312025-04-30T06:26:35.816ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
74322025-04-30T06:26:35.820ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "853082b9-3d04-498a-b48c-a2abd6895d69", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74332025-04-30T06:26:35.820ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
74342025-04-30T06:26:35.824ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9900c0db-8d4d-4103-ac33-2d752320b7f3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74352025-04-30T06:26:35.824ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
74362025-04-30T06:26:35.828ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db059772-4f12-431d-b6dc-c2bad4041f08", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74372025-04-30T06:26:35.829ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
74382025-04-30T06:26:35.834ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "591c0e69-1e6b-4344-bc67-c509ab8ad5c3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74392025-04-30T06:26:35.834ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
74402025-04-30T06:26:35.837ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff0a05f5-457d-47f6-bd01-bf2a034b26e6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74412025-04-30T06:26:35.837ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
74422025-04-30T06:26:35.840ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be080fff-efd6-4f40-9698-20f1f50cdbdd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74432025-04-30T06:26:35.840ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
74442025-04-30T06:26:35.844ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e70ee8d3-d4bd-4eff-b545-d8d8a6457f3d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74452025-04-30T06:26:35.844ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
74462025-04-30T06:26:35.844ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
74472025-04-30T06:26:35.846ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d2ac69e-1565-4208-811a-d62a5c994f58", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74482025-04-30T06:26:35.847ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
74492025-04-30T06:26:35.848ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0902d61-2ec7-4766-aa12-4a2982d361ae", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74502025-04-30T06:26:35.848ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
74512025-04-30T06:26:35.849ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a99a985b-c5d0-4e72-8eb2-543c47b96c20", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74522025-04-30T06:26:35.849ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
74532025-04-30T06:26:35.851ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0e58234-754b-4e19-8d6f-6a895a837975", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74542025-04-30T06:26:35.851ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
74552025-04-30T06:26:35.854ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f56d4a3d-df80-4b87-a2bf-ea328864b58f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74562025-04-30T06:26:35.854ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
74572025-04-30T06:26:35.858ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d569796-0077-4c2d-a041-0508c0116ddf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74582025-04-30T06:26:35.858ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
74592025-04-30T06:26:35.859ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3544431-a53f-4141-82fe-fdec24664901", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74602025-04-30T06:26:35.860ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
74612025-04-30T06:26:35.862ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1175cf36-adb2-4782-8b13-ed9420ad19ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74622025-04-30T06:26:35.862ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
74632025-04-30T06:26:35.865ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "af0218c0-d54e-41d2-b372-6512114a6b45", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74642025-04-30T06:26:35.866ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
74652025-04-30T06:26:35.868ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "340dc1dc-2270-4241-bc1a-2109a9a56571", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74662025-04-30T06:26:35.868ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
74672025-04-30T06:26:35.871ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "185c27f3-9ed2-4902-ae4b-5703b5c4d446", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74682025-04-30T06:26:35.871ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
74692025-04-30T06:26:35.872ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "776ccba1-7b52-4768-8140-4b73852c3b22", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74702025-04-30T06:26:35.872ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
74712025-04-30T06:26:35.873ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25142454-db05-4bb6-8a0b-133fb1cc0391", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74722025-04-30T06:26:35.873ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
74732025-04-30T06:26:35.874ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f84a52be-ff6b-4e13-b198-6505a9c02816", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
74742025-04-30T06:26:35.874ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
74752025-04-30T06:26:35.874ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74762025-04-30T06:26:35.874ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74772025-04-30T06:26:35.874ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74782025-04-30T06:26:35.875ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74792025-04-30T06:26:35.878ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74802025-04-30T06:26:35.886ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
74812025-04-30T06:26:35.887ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
74822025-04-30T06:26:35.887ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74832025-04-30T06:26:35.887ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74842025-04-30T06:26:35.887ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74852025-04-30T06:26:35.888ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74862025-04-30T06:26:35.891ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74872025-04-30T06:26:35.900ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
74882025-04-30T06:26:35.900ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74892025-04-30T06:26:35.900ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74902025-04-30T06:26:35.900ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74912025-04-30T06:26:35.901ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74922025-04-30T06:26:35.901ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74932025-04-30T06:26:35.901ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74942025-04-30T06:26:35.901ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
74952025-04-30T06:26:35.902ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74962025-04-30T06:26:35.959ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
74972025-04-30T06:26:35.959ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
74982025-04-30T06:26:35.960ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74992025-04-30T06:26:35.975ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75002025-04-30T06:26:35.975ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75012025-04-30T06:26:35.975ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
75022025-04-30T06:26:35.976ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75032025-04-30T06:26:35.976ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:56870 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
75042025-04-30T06:26:35.976ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
75052025-04-30T06:26:35.976ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
75062025-04-30T06:26:35.977ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
75072025-04-30T06:26:35.978ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
75082025-04-30T06:26:35.987ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:56870 req_id = 5961f673-61d8-42aa-88a3-044ab81e718f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
75092025-04-30T06:26:35.989ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1789 local_addr = [::1]:44559 method = GET remote_addr = [::1]:56870 req_id = 5961f673-61d8-42aa-88a3-044ab81e718f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
75102025-04-30T06:26:35.990ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75112025-04-30T06:26:35.990ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
75122025-04-30T06:26:35.992ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75132025-04-30T06:26:35.992ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
75142025-04-30T06:26:35.992ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
75152025-04-30T06:26:35.993ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5961f673-61d8-42aa-88a3-044ab81e718f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
75162025-04-30T06:26:35.996ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
75172025-04-30T06:26:35.996ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
75182025-04-30T06:26:35.996ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
75192025-04-30T06:26:36.001ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
75202025-04-30T06:26:36.001ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75212025-04-30T06:26:36.001ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75222025-04-30T06:26:36.001ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75232025-04-30T06:26:36.002ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75242025-04-30T06:26:36.009ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
75252025-04-30T06:26:36.009ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
75262025-04-30T06:26:36.010ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75272025-04-30T06:26:36.024ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
75282025-04-30T06:26:36.024ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75292025-04-30T06:26:36.024ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75302025-04-30T06:26:36.029ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75312025-04-30T06:26:36.049ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75322025-04-30T06:26:36.049ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75332025-04-30T06:26:36.064ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75342025-04-30T06:26:36.065ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75352025-04-30T06:26:36.065ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75362025-04-30T06:26:36.065ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75372025-04-30T06:26:36.084ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
75382025-04-30T06:26:36.098ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
75392025-04-30T06:26:36.098ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
75402025-04-30T06:26:36.098ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 578.541647ms iteration = 23 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
75412025-04-30T06:26:36.098ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 24 reason = Signaled
75422025-04-30T06:26:36.098ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
75432025-04-30T06:26:36.098ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75442025-04-30T06:26:36.098ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75452025-04-30T06:26:36.098ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75462025-04-30T06:26:36.103ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75472025-04-30T06:26:36.106ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75482025-04-30T06:26:36.111ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
75492025-04-30T06:26:36.111ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75502025-04-30T06:26:36.111ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75512025-04-30T06:26:36.111ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75522025-04-30T06:26:36.113ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
75532025-04-30T06:26:36.116ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
75542025-04-30T06:26:36.116ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
75552025-04-30T06:26:36.118ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 41 21 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 A!..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60456 req_id = 7dff5a48-707b-4a6a-9494-47764fd9ef9b
75562025-04-30T06:26:36.118ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16673,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60456 req_id = 7dff5a48-707b-4a6a-9494-47764fd9ef9b
75572025-04-30T06:26:36.118ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
75582025-04-30T06:26:36.118ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
75592025-04-30T06:26:36.118ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60456 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7dff5a48-707b-4a6a-9494-47764fd9ef9b
75602025-04-30T06:26:36.118ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3260, tv_nsec: 917779348 } })
75612025-04-30T06:26:36.118ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
75622025-04-30T06:26:36.119ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 86 cb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62922 req_id = 80f90406-9955-4299-8575-feda3d3913cb
75632025-04-30T06:26:36.119ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34507,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62922 req_id = 80f90406-9955-4299-8575-feda3d3913cb
75642025-04-30T06:26:36.119ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
75652025-04-30T06:26:36.119ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62922 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 80f90406-9955-4299-8575-feda3d3913cb
75662025-04-30T06:26:36.178ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
75672025-04-30T06:26:36.179ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75682025-04-30T06:26:36.179ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75692025-04-30T06:26:36.179ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75702025-04-30T06:26:36.179ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
75712025-04-30T06:26:36.180ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75722025-04-30T06:26:36.182ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 50 fd 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c P............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52529 req_id = 425742e3-ef09-49c2-96f1-d3df305dc431
75732025-04-30T06:26:36.182ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20733,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52529 req_id = 425742e3-ef09-49c2-96f1-d3df305dc431
75742025-04-30T06:26:36.182ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
75752025-04-30T06:26:36.182ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 20733, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52529 req_id = 425742e3-ef09-49c2-96f1-d3df305dc431
75762025-04-30T06:26:36.182ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
75772025-04-30T06:26:36.183ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
75782025-04-30T06:26:36.183ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
75792025-04-30T06:26:36.183ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75802025-04-30T06:26:36.190ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75812025-04-30T06:26:36.190ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75822025-04-30T06:26:36.190ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
75832025-04-30T06:26:36.191ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75842025-04-30T06:26:36.193ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dc0376d-8f8e-4ad7-9d7f-c2f88311d0c8", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
75852025-04-30T06:26:36.193ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
75862025-04-30T06:26:36.194ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75872025-04-30T06:26:36.194ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
75882025-04-30T06:26:36.196ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75892025-04-30T06:26:36.200ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1d3ea62-16a9-41de-8d7f-f36dc815ac22", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
75902025-04-30T06:26:36.200ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
75912025-04-30T06:26:36.204ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a006c09f-597c-4241-8beb-80b5e31c7de1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
75922025-04-30T06:26:36.204ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
75932025-04-30T06:26:36.214ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f55d568-a9cd-4f3e-845c-5eb5b6ee9dc3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
75942025-04-30T06:26:36.214ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
75952025-04-30T06:26:36.219ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e1ba08d-3379-4f11-a3ae-1c4ff4a0cd44", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
75962025-04-30T06:26:36.219ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
75972025-04-30T06:26:36.226ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "290b99ee-54fa-4eef-bafa-b5645d930645", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
75982025-04-30T06:26:36.226ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
75992025-04-30T06:26:36.230ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
76002025-04-30T06:26:36.234ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99174681-f422-47f4-b20c-f3d8c664f758", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76012025-04-30T06:26:36.234ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
76022025-04-30T06:26:36.238ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e99857f-7d98-4ff5-99df-ce040b937e6b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76032025-04-30T06:26:36.239ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
76042025-04-30T06:26:36.240ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9053576c-9fe2-4686-a159-c252cce068e2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76052025-04-30T06:26:36.241ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
76062025-04-30T06:26:36.242ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28a64197-a786-416d-b392-916c8084d35c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76072025-04-30T06:26:36.243ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
76082025-04-30T06:26:36.244ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d76fd00b-ff25-4751-86fc-d8c189f0f511", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76092025-04-30T06:26:36.252ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
76102025-04-30T06:26:36.253ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a041677b-b36f-4fd2-bec4-06e138bba9cc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76112025-04-30T06:26:36.253ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
76122025-04-30T06:26:36.255ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1b2f72e-1b60-4af4-889a-ec3c2e7b7ae9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76132025-04-30T06:26:36.255ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
76142025-04-30T06:26:36.257ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c50398c-ae93-4257-9f56-6d8e1197e744", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76152025-04-30T06:26:36.260ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
76162025-04-30T06:26:36.264ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74e2ea72-143e-4dda-abaf-61845215f131", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76172025-04-30T06:26:36.265ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
76182025-04-30T06:26:36.269ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e5811e6-f02a-4f5c-bcfc-a72e7f7b32aa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76192025-04-30T06:26:36.269ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
76202025-04-30T06:26:36.273ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dae0a299-cf9e-412d-b48a-166e9235d69d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76212025-04-30T06:26:36.275ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
76222025-04-30T06:26:36.279ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7941df9-e1ba-4c40-8c91-2a243fbae3d3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76232025-04-30T06:26:36.279ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
76242025-04-30T06:26:36.282ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76252025-04-30T06:26:36.282ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76262025-04-30T06:26:36.282ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76272025-04-30T06:26:36.282ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
76282025-04-30T06:26:36.284ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76292025-04-30T06:26:36.286ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cec9eb2-1f89-4ea9-aa4c-cad00e58813d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76302025-04-30T06:26:36.287ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
76312025-04-30T06:26:36.287ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76322025-04-30T06:26:36.298ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76332025-04-30T06:26:36.298ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76342025-04-30T06:26:36.298ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
76352025-04-30T06:26:36.303ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76362025-04-30T06:26:36.307ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f21f8b2b-9416-4f19-b753-8872e028dbe6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76372025-04-30T06:26:36.307ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
76382025-04-30T06:26:36.307ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76392025-04-30T06:26:36.307ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
76402025-04-30T06:26:36.308ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76412025-04-30T06:26:36.309ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e70643b-e1c9-495c-99f4-f8f8f7f26da9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76422025-04-30T06:26:36.309ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
76432025-04-30T06:26:36.310ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ea162d5-0851-4a76-8c97-1474620312d3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76442025-04-30T06:26:36.310ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
76452025-04-30T06:26:36.311ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a06fa9f1-8d61-4c85-85af-3fa1842a151d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76462025-04-30T06:26:36.311ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
76472025-04-30T06:26:36.312ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db3a10b3-3310-4032-81bb-d4d5c6c0c4bf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76482025-04-30T06:26:36.312ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
76492025-04-30T06:26:36.315ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f31bdf4-52da-4e86-bb66-76892299970d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76502025-04-30T06:26:36.315ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
76512025-04-30T06:26:36.322ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53a032c4-47ce-493c-b6fb-c55e2ab7061c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76522025-04-30T06:26:36.323ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
76532025-04-30T06:26:36.327ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "23a180d6-60aa-4d5a-a040-f27f7921de8d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:35 GMT"} })
76542025-04-30T06:26:36.328ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
76552025-04-30T06:26:36.332ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "585d2031-f1d0-4c39-ae55-dd727b3fcb07", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76562025-04-30T06:26:36.333ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
76572025-04-30T06:26:36.346ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
76582025-04-30T06:26:36.355ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "707a0d13-5190-4938-81ca-416f95d8e073", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76592025-04-30T06:26:36.360ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
76602025-04-30T06:26:36.365ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c84024a-c5d0-4109-ba8d-296011d2a1ae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76612025-04-30T06:26:36.365ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
76622025-04-30T06:26:36.367ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "888e2fb8-3951-4276-a6a8-5eae8b6bf8f2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76632025-04-30T06:26:36.368ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
76642025-04-30T06:26:36.369ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4173ec8b-939e-4246-815c-c6834158dfc4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76652025-04-30T06:26:36.370ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
76662025-04-30T06:26:36.372ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fbc466c-49f2-48aa-b434-38766d610882", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76672025-04-30T06:26:36.372ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
76682025-04-30T06:26:36.374ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cb35f7b-730e-48fc-a12b-1e78a0d4386d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76692025-04-30T06:26:36.374ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
76702025-04-30T06:26:36.378ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64bc8dac-21e8-445d-bca8-b8ff38d9c295", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76712025-04-30T06:26:36.379ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
76722025-04-30T06:26:36.395ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97d70e9b-401a-4f32-a5c3-e3950b83c5ba", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76732025-04-30T06:26:36.395ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
76742025-04-30T06:26:36.404ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76752025-04-30T06:26:36.404ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76762025-04-30T06:26:36.404ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76772025-04-30T06:26:36.404ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
76782025-04-30T06:26:36.406ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76792025-04-30T06:26:36.408ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61f12301-f2f9-44f4-839a-b47f66ad1d9f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76802025-04-30T06:26:36.409ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
76812025-04-30T06:26:36.409ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76822025-04-30T06:26:36.441ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76832025-04-30T06:26:36.442ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76842025-04-30T06:26:36.442ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
76852025-04-30T06:26:36.444ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76862025-04-30T06:26:36.447ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76872025-04-30T06:26:36.447ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
76882025-04-30T06:26:36.448ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76892025-04-30T06:26:36.455ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e0e051e-d534-42b7-933b-1c7befdfe573", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76902025-04-30T06:26:36.456ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
76912025-04-30T06:26:36.466ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff4127cf-7539-4250-b410-621d2e20274e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76922025-04-30T06:26:36.466ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
76932025-04-30T06:26:36.483ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bf7fb22-8a0a-4987-a0ba-be5c5469fa8f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76942025-04-30T06:26:36.483ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
76952025-04-30T06:26:36.492ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
76962025-04-30T06:26:36.494ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "698711c0-150a-4c43-949d-6f0659ad0f54", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76972025-04-30T06:26:36.494ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
76982025-04-30T06:26:36.496ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19a48182-2094-4895-bc75-a51731fcdcaf", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
76992025-04-30T06:26:36.496ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
77002025-04-30T06:26:36.497ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a125fbd-350c-4ba0-8d3f-94ec4cdf5ad1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
77012025-04-30T06:26:36.497ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
77022025-04-30T06:26:36.499ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bcd45c4-bf09-481b-a232-8237cbea0146", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
77032025-04-30T06:26:36.503ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
77042025-04-30T06:26:36.504ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eac25be-8958-4b76-a3ba-2e589f167255", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
77052025-04-30T06:26:36.504ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
77062025-04-30T06:26:36.504ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77072025-04-30T06:26:36.504ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77082025-04-30T06:26:36.504ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77092025-04-30T06:26:36.505ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77102025-04-30T06:26:36.519ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77112025-04-30T06:26:36.526ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
77122025-04-30T06:26:36.526ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
77132025-04-30T06:26:36.527ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77142025-04-30T06:26:36.527ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77152025-04-30T06:26:36.527ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77162025-04-30T06:26:36.528ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77172025-04-30T06:26:36.532ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77182025-04-30T06:26:36.537ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
77192025-04-30T06:26:36.537ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77202025-04-30T06:26:36.537ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77212025-04-30T06:26:36.538ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77222025-04-30T06:26:36.548ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77232025-04-30T06:26:36.548ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77242025-04-30T06:26:36.548ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77252025-04-30T06:26:36.548ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
77262025-04-30T06:26:36.548ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77272025-04-30T06:26:36.604ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
77282025-04-30T06:26:36.604ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
77292025-04-30T06:26:36.640ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77302025-04-30T06:26:36.658ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77312025-04-30T06:26:36.662ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77322025-04-30T06:26:36.662ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
77332025-04-30T06:26:36.670ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77342025-04-30T06:26:36.671ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:53824 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
77352025-04-30T06:26:36.672ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:53824 req_id = cf189997-8632-4948-8d90-fcf252fbc951 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
77362025-04-30T06:26:36.673ZINFOtest_disk_expunge (dropshot): request completed latency_us = 832 local_addr = [::1]:44559 method = GET remote_addr = [::1]:53824 req_id = cf189997-8632-4948-8d90-fcf252fbc951 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
77372025-04-30T06:26:36.675ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf189997-8632-4948-8d90-fcf252fbc951", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
77382025-04-30T06:26:36.677ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
77392025-04-30T06:26:36.677ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
77402025-04-30T06:26:36.677ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
77412025-04-30T06:26:36.679ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
77422025-04-30T06:26:36.679ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77432025-04-30T06:26:36.679ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77442025-04-30T06:26:36.679ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77452025-04-30T06:26:36.680ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77462025-04-30T06:26:36.680ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77472025-04-30T06:26:36.680ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
77482025-04-30T06:26:36.681ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77492025-04-30T06:26:36.687ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77502025-04-30T06:26:36.690ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
77512025-04-30T06:26:36.691ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77522025-04-30T06:26:36.691ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77532025-04-30T06:26:36.692ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77542025-04-30T06:26:36.758ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
77552025-04-30T06:26:36.759ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
77562025-04-30T06:26:36.759ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 661.146773ms iteration = 24 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
77572025-04-30T06:26:36.760ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 25 reason = Signaled
77582025-04-30T06:26:36.760ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
77592025-04-30T06:26:36.760ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77602025-04-30T06:26:36.760ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77612025-04-30T06:26:36.760ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77622025-04-30T06:26:36.762ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77632025-04-30T06:26:36.763ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
77642025-04-30T06:26:36.771ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77652025-04-30T06:26:36.794ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
77662025-04-30T06:26:36.799ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
77672025-04-30T06:26:36.799ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
77682025-04-30T06:26:36.800ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
77692025-04-30T06:26:36.805ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
77702025-04-30T06:26:36.805ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
77712025-04-30T06:26:36.807ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0d 77 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .w..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52497 req_id = 0aececb1-b407-4e6e-a921-79827d3a838c
77722025-04-30T06:26:36.807ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3447,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52497 req_id = 0aececb1-b407-4e6e-a921-79827d3a838c
77732025-04-30T06:26:36.807ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
77742025-04-30T06:26:36.807ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
77752025-04-30T06:26:36.807ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52497 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0aececb1-b407-4e6e-a921-79827d3a838c
77762025-04-30T06:26:36.808ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3261, tv_nsec: 606985475 } })
77772025-04-30T06:26:36.813ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
77782025-04-30T06:26:36.813ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 3d c2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e =...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64691 req_id = 617815a5-0a79-4dd0-8c9a-1e1727059065
77792025-04-30T06:26:36.814ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15810,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64691 req_id = 617815a5-0a79-4dd0-8c9a-1e1727059065
77802025-04-30T06:26:36.814ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
77812025-04-30T06:26:36.814ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64691 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 617815a5-0a79-4dd0-8c9a-1e1727059065
77822025-04-30T06:26:36.814ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77832025-04-30T06:26:36.814ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77842025-04-30T06:26:36.814ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77852025-04-30T06:26:36.814ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
77862025-04-30T06:26:36.815ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77872025-04-30T06:26:36.873ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
77882025-04-30T06:26:36.881ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
77892025-04-30T06:26:36.882ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
77902025-04-30T06:26:36.884ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 16b067a9-996e-4b90-bbe0-c9ce76b9593c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
77912025-04-30T06:26:36.889ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e0 6c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .l..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52363 req_id = 562cef09-b67c-4810-93b8-0e63d8199a40
77922025-04-30T06:26:36.889ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57452,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52363 req_id = 562cef09-b67c-4810-93b8-0e63d8199a40
77932025-04-30T06:26:36.889ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
77942025-04-30T06:26:36.889ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 57452, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52363 req_id = 562cef09-b67c-4810-93b8-0e63d8199a40
77952025-04-30T06:26:36.890ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 16b067a9-996e-4b90-bbe0-c9ce76b9593c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
77962025-04-30T06:26:36.890ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 16b067a9-996e-4b90-bbe0-c9ce76b9593c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
77972025-04-30T06:26:36.892ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 16b067a9-996e-4b90-bbe0-c9ce76b9593c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
77982025-04-30T06:26:36.897ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77992025-04-30T06:26:36.902ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78002025-04-30T06:26:36.902ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78012025-04-30T06:26:36.902ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
78022025-04-30T06:26:36.903ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78032025-04-30T06:26:36.903ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
78042025-04-30T06:26:36.903ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
78052025-04-30T06:26:36.903ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
78062025-04-30T06:26:36.908ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 24175 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 16b067a9-996e-4b90-bbe0-c9ce76b9593c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
78072025-04-30T06:26:36.910ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "16b067a9-996e-4b90-bbe0-c9ce76b9593c", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
78082025-04-30T06:26:36.910ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
78092025-04-30T06:26:36.910ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78102025-04-30T06:26:36.910ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
78112025-04-30T06:26:36.911ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78122025-04-30T06:26:36.911ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = daa90503-4532-4565-b8d7-752bdc6f4733 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
78132025-04-30T06:26:36.911ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = daa90503-4532-4565-b8d7-752bdc6f4733 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
78142025-04-30T06:26:36.911ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = daa90503-4532-4565-b8d7-752bdc6f4733 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
78152025-04-30T06:26:36.912ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = daa90503-4532-4565-b8d7-752bdc6f4733 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
78162025-04-30T06:26:36.915ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9ff77c9-445f-4d13-8aba-4643a0340c31", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
78172025-04-30T06:26:36.923ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
78182025-04-30T06:26:36.924ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 12755 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = daa90503-4532-4565-b8d7-752bdc6f4733 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
78192025-04-30T06:26:36.925ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "daa90503-4532-4565-b8d7-752bdc6f4733", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
78202025-04-30T06:26:36.925ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
78212025-04-30T06:26:36.925ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78222025-04-30T06:26:36.925ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
78232025-04-30T06:26:36.930ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "085cd7ae-aedb-4f91-9a1e-47a2399de9b2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
78242025-04-30T06:26:36.930ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
78252025-04-30T06:26:36.933ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0adefb64-e97b-49c7-ade8-9ee058533cd1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
78262025-04-30T06:26:36.933ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
78272025-04-30T06:26:36.941ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec07bbf6-8ef8-4928-8414-b2d6dd224d26", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
78282025-04-30T06:26:36.942ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
78292025-04-30T06:26:36.955ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33271f3b-1980-444d-a5b3-281a787f43a2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
78302025-04-30T06:26:36.956ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
78312025-04-30T06:26:36.956ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78322025-04-30T06:26:36.957ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78332025-04-30T06:26:36.958ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
78342025-04-30T06:26:36.959ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
78352025-04-30T06:26:36.967ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
78362025-04-30T06:26:36.967ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
78372025-04-30T06:26:36.968ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
78382025-04-30T06:26:36.971ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
78392025-04-30T06:26:36.971ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
78402025-04-30T06:26:36.972ZINFOtest_disk_expunge (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = 127.0.0.1:38728 remote_addr = 127.0.0.1:42469
78412025-04-30T06:26:36.972ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
78422025-04-30T06:26:36.973ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78432025-04-30T06:26:36.973ZTRACtest_disk_expunge (dropshot): incoming request local_addr = 127.0.0.1:38728 method = GET remote_addr = 127.0.0.1:42469 req_id = b08cfc39-d7db-4564-ae31-630a3918f548 uri = /78121327-166d-4e33-935c-82ab8635bf33
78442025-04-30T06:26:36.979ZINFOtest_disk_expunge (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 5168 local_addr = 127.0.0.1:38728 method = GET remote_addr = 127.0.0.1:42469 req_id = b08cfc39-d7db-4564-ae31-630a3918f548 response_code = 200 uri = /78121327-166d-4e33-935c-82ab8635bf33
78452025-04-30T06:26:36.979ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
78462025-04-30T06:26:36.995ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
78472025-04-30T06:26:36.995ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
78482025-04-30T06:26:36.995ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
78492025-04-30T06:26:36.995ZDEBGtest_disk_expunge (results-sink): inserting 17 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78502025-04-30T06:26:36.995ZDEBGtest_disk_expunge (clickhouse-client): unrolling 17 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
78512025-04-30T06:26:36.995ZDEBGtest_disk_expunge (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
78522025-04-30T06:26:36.996ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('integration_target:integration_metric') FORMAT Native;
78532025-04-30T06:26:36.999ZTRACtest_disk_expunge (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
78542025-04-30T06:26:37.001ZDEBGtest_disk_expunge (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:26:36.995861040Z } timeseries_name = http_service:request_latency_histogram
78552025-04-30T06:26:37.007ZDEBGtest_disk_expunge (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
78562025-04-30T06:26:37.007ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
78572025-04-30T06:26:37.016ZTRACtest_disk_expunge (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
78582025-04-30T06:26:37.017ZDEBGtest_disk_expunge (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:26:37.007337947Z } timeseries_name = collection_target:cpus_provisioned
78592025-04-30T06:26:37.018ZDEBGtest_disk_expunge (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
78602025-04-30T06:26:37.018ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
78612025-04-30T06:26:37.020ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78622025-04-30T06:26:37.021ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78632025-04-30T06:26:37.022ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78642025-04-30T06:26:37.022ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
78652025-04-30T06:26:37.023ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78662025-04-30T06:26:37.024ZTRACtest_disk_expunge (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
78672025-04-30T06:26:37.024ZDEBGtest_disk_expunge (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:26:37.018211401Z } timeseries_name = collection_target:ram_provisioned
78682025-04-30T06:26:37.025ZDEBGtest_disk_expunge (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
78692025-04-30T06:26:37.025ZTRACtest_disk_expunge (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'collection_target:ram_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
78702025-04-30T06:26:37.032ZTRACtest_disk_expunge (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
78712025-04-30T06:26:37.032ZDEBGtest_disk_expunge (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:26:37.025737486Z } timeseries_name = collection_target:virtual_disk_space_provisioned
78722025-04-30T06:26:37.042ZDEBGtest_disk_expunge (clickhouse-client): inserting 4 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
78732025-04-30T06:26:37.042ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 6 n_rows = 4 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
78742025-04-30T06:26:37.044ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f6e3b09-da60-4354-be10-3c3c52a5a8bf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
78752025-04-30T06:26:37.044ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
78762025-04-30T06:26:37.054ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78772025-04-30T06:26:37.058ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78782025-04-30T06:26:37.077ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78792025-04-30T06:26:37.114ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78802025-04-30T06:26:37.114ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78812025-04-30T06:26:37.114ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
78822025-04-30T06:26:37.115ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78832025-04-30T06:26:37.117ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 10 sql = INSERT INTO oximeter.fields_string FORMAT Native
78842025-04-30T06:26:37.120ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
78852025-04-30T06:26:37.123ZDEBGtest_disk_expunge (results-sink-cluster): inserting 17 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
78862025-04-30T06:26:37.138ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78872025-04-30T06:26:37.149ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78882025-04-30T06:26:37.149ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78892025-04-30T06:26:37.149ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78902025-04-30T06:26:37.150ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78912025-04-30T06:26:37.150ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
78922025-04-30T06:26:37.151ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78932025-04-30T06:26:37.155ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 10 table_name = fields_string
78942025-04-30T06:26:37.155ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
78952025-04-30T06:26:37.170ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 5 table_name = fields_u16
78962025-04-30T06:26:37.170ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 17 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
78972025-04-30T06:26:37.180ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 17 table_name = fields_uuid
78982025-04-30T06:26:37.180ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
78992025-04-30T06:26:37.182ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dd21617-3f50-48c9-a196-5e4eb1d6eb64", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79002025-04-30T06:26:37.182ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
79012025-04-30T06:26:37.192ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 5 table_name = measurements_histogramu64
79022025-04-30T06:26:37.192ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
79032025-04-30T06:26:37.194ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36b8dce9-c893-4176-88f7-ae519debc269", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79042025-04-30T06:26:37.194ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
79052025-04-30T06:26:37.197ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6ea63ca-423b-47ae-ab32-ddc791498e5f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79062025-04-30T06:26:37.198ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
79072025-04-30T06:26:37.199ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 12 table_name = measurements_i64
79082025-04-30T06:26:37.199ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
79092025-04-30T06:26:37.202ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5996d465-333e-454a-a481-88fb5be76bbb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79102025-04-30T06:26:37.202ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
79112025-04-30T06:26:37.204ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2cb772c-5281-4cb7-b301-a0be9417e280", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79122025-04-30T06:26:37.205ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
79132025-04-30T06:26:37.210ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "154add2c-8e6a-417a-bd67-b8bdf211e33d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79142025-04-30T06:26:37.210ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
79152025-04-30T06:26:37.216ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
79162025-04-30T06:26:37.216ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4046ae4-95cd-4a1f-83f6-0c5218b03d17", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79172025-04-30T06:26:37.216ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
79182025-04-30T06:26:37.221ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be34fe74-f61b-46d6-8d37-311669787cd1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79192025-04-30T06:26:37.222ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
79202025-04-30T06:26:37.234ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
79212025-04-30T06:26:37.246ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d89b335-43a6-4ec9-ae57-05b1899e6ef7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79222025-04-30T06:26:37.246ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
79232025-04-30T06:26:37.248ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c449792-ef17-471c-8104-ea80a596e0fe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79242025-04-30T06:26:37.249ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
79252025-04-30T06:26:37.251ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca451cfe-0088-4977-8596-759ae03aba4b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79262025-04-30T06:26:37.251ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
79272025-04-30T06:26:37.253ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "284d30d3-afec-4c3c-a25f-638b92660bb5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79282025-04-30T06:26:37.253ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
79292025-04-30T06:26:37.257ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fe436b2-62cc-489e-b47e-efc8caea41e5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79302025-04-30T06:26:37.257ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
79312025-04-30T06:26:37.258ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c095e0d7-ca52-406b-ba20-0ef77a0e2fb7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79322025-04-30T06:26:37.259ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
79332025-04-30T06:26:37.260ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "53765103-a627-4ad4-8704-55ba9466ccf1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79342025-04-30T06:26:37.261ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
79352025-04-30T06:26:37.262ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14921a64-d53b-4d27-aff5-0f1c011efc21", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79362025-04-30T06:26:37.263ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
79372025-04-30T06:26:37.264ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c26da61c-2043-43b5-8c79-de900b27559c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79382025-04-30T06:26:37.264ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
79392025-04-30T06:26:37.266ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aea65952-cbe2-4b45-ab6f-e531e0f5a21c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79402025-04-30T06:26:37.266ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
79412025-04-30T06:26:37.267ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79422025-04-30T06:26:37.267ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79432025-04-30T06:26:37.267ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79442025-04-30T06:26:37.267ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
79452025-04-30T06:26:37.268ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79462025-04-30T06:26:37.270ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa74f024-22ba-484b-9bac-a783793956f9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79472025-04-30T06:26:37.270ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
79482025-04-30T06:26:37.270ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79492025-04-30T06:26:37.276ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79502025-04-30T06:26:37.276ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79512025-04-30T06:26:37.277ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
79522025-04-30T06:26:37.277ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79532025-04-30T06:26:37.280ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79542025-04-30T06:26:37.280ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
79552025-04-30T06:26:37.280ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79562025-04-30T06:26:37.281ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a6fe40a-d78c-43a9-8298-69e1a9bc0f41", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79572025-04-30T06:26:37.281ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
79582025-04-30T06:26:37.284ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19fe9bee-fa12-44e7-985b-56e93bcccdc9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79592025-04-30T06:26:37.288ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
79602025-04-30T06:26:37.293ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1adf4dd5-c52b-44cb-9f9f-7183bd617704", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79612025-04-30T06:26:37.294ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
79622025-04-30T06:26:37.302ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2155f499-112b-4b69-bcd0-033c9d82099d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79632025-04-30T06:26:37.303ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
79642025-04-30T06:26:37.303ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8ad96c4-c745-44c9-aaca-7aef5d594d93", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79652025-04-30T06:26:37.303ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
79662025-04-30T06:26:37.307ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f85e4395-a5a1-4228-8dee-8a8760e09cd8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79672025-04-30T06:26:37.309ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
79682025-04-30T06:26:37.309ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0361ea57-f3b6-4084-adb4-a0bd95f6697b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79692025-04-30T06:26:37.309ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
79702025-04-30T06:26:37.312ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d22ce42-739a-4c9c-9769-e6ad080d190f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79712025-04-30T06:26:37.312ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
79722025-04-30T06:26:37.313ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0362ed05-6001-4ab5-82bc-2758f3de3b74", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79732025-04-30T06:26:37.313ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
79742025-04-30T06:26:37.318ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bae7e7bf-faf7-4fc3-b240-da81dbd55fae", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79752025-04-30T06:26:37.318ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
79762025-04-30T06:26:37.322ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ab2964a-3637-4bb1-adb0-f4a341fba46e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79772025-04-30T06:26:37.322ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
79782025-04-30T06:26:37.325ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
79792025-04-30T06:26:37.325ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bbfe79c-aeb9-4fe9-b786-6e5865912714", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:36 GMT"} })
79802025-04-30T06:26:37.326ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
79812025-04-30T06:26:37.329ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68de0417-29d7-40f8-8cbd-97df2394a758", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
79822025-04-30T06:26:37.329ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
79832025-04-30T06:26:37.333ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0576dac2-4de7-46f0-a565-f11038e8dfce", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
79842025-04-30T06:26:37.333ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
79852025-04-30T06:26:37.336ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a030311-25e0-45cd-839c-5eef90fe2a49", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
79862025-04-30T06:26:37.336ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
79872025-04-30T06:26:37.338ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1941c2ce-ba0b-46e2-9da5-ea2be54df5cd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
79882025-04-30T06:26:37.338ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
79892025-04-30T06:26:37.339ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "725e510e-a0b8-4e3d-994f-2bf0de6ec950", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
79902025-04-30T06:26:37.339ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
79912025-04-30T06:26:37.341ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d274a452-2471-4c5b-b826-6bac3ce87c08", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
79922025-04-30T06:26:37.342ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
79932025-04-30T06:26:37.344ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb2b34fa-b940-45a3-bd00-eed48d0bfde1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
79942025-04-30T06:26:37.344ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
79952025-04-30T06:26:37.348ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a4d8c03-1be0-46f9-8987-2cf92980b08e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
79962025-04-30T06:26:37.348ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
79972025-04-30T06:26:37.348ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79982025-04-30T06:26:37.348ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
79992025-04-30T06:26:37.348ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80002025-04-30T06:26:37.349ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80012025-04-30T06:26:37.376ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80022025-04-30T06:26:37.376ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80032025-04-30T06:26:37.376ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80042025-04-30T06:26:37.376ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
80052025-04-30T06:26:37.377ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80062025-04-30T06:26:37.378ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80072025-04-30T06:26:37.382ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
80082025-04-30T06:26:37.383ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
80092025-04-30T06:26:37.383ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80102025-04-30T06:26:37.383ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80112025-04-30T06:26:37.383ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80122025-04-30T06:26:37.383ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80132025-04-30T06:26:37.386ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80142025-04-30T06:26:37.399ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
80152025-04-30T06:26:37.399ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80162025-04-30T06:26:37.399ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80172025-04-30T06:26:37.400ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80182025-04-30T06:26:37.404ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80192025-04-30T06:26:37.415ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80202025-04-30T06:26:37.415ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80212025-04-30T06:26:37.415ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
80222025-04-30T06:26:37.416ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80232025-04-30T06:26:37.508ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
80242025-04-30T06:26:37.508ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
80252025-04-30T06:26:37.510ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:40319 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
80262025-04-30T06:26:37.511ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80272025-04-30T06:26:37.512ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
80282025-04-30T06:26:37.514ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80292025-04-30T06:26:37.514ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:40319 req_id = c3d6a420-53b9-4d96-9193-0485592792d4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
80302025-04-30T06:26:37.515ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1038 local_addr = [::1]:44559 method = GET remote_addr = [::1]:40319 req_id = c3d6a420-53b9-4d96-9193-0485592792d4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
80312025-04-30T06:26:37.516ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3d6a420-53b9-4d96-9193-0485592792d4", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
80322025-04-30T06:26:37.519ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
80332025-04-30T06:26:37.519ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
80342025-04-30T06:26:37.519ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
80352025-04-30T06:26:37.522ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
80362025-04-30T06:26:37.524ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80372025-04-30T06:26:37.524ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80382025-04-30T06:26:37.524ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80392025-04-30T06:26:37.528ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80402025-04-30T06:26:37.532ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80412025-04-30T06:26:37.536ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
80422025-04-30T06:26:37.537ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80432025-04-30T06:26:37.537ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80442025-04-30T06:26:37.537ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80452025-04-30T06:26:37.572ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
80462025-04-30T06:26:37.575ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
80472025-04-30T06:26:37.576ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
80482025-04-30T06:26:37.576ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 816.334944ms iteration = 25 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
80492025-04-30T06:26:37.576ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 26 reason = Signaled
80502025-04-30T06:26:37.576ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
80512025-04-30T06:26:37.576ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80522025-04-30T06:26:37.576ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80532025-04-30T06:26:37.576ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80542025-04-30T06:26:37.577ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80552025-04-30T06:26:37.579ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80562025-04-30T06:26:37.586ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
80572025-04-30T06:26:37.586ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80582025-04-30T06:26:37.586ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80592025-04-30T06:26:37.586ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80602025-04-30T06:26:37.589ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
80612025-04-30T06:26:37.589ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
80622025-04-30T06:26:37.593ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 7e 71 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ~q..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65460 req_id = 82c04000-eb9a-4b79-8bd5-4347039e7f23
80632025-04-30T06:26:37.593ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32369,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65460 req_id = 82c04000-eb9a-4b79-8bd5-4347039e7f23
80642025-04-30T06:26:37.593ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
80652025-04-30T06:26:37.594ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
80662025-04-30T06:26:37.594ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65460 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 82c04000-eb9a-4b79-8bd5-4347039e7f23
80672025-04-30T06:26:37.594ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3262, tv_nsec: 393630104 } })
80682025-04-30T06:26:37.594ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
80692025-04-30T06:26:37.596ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8a fc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54662 req_id = 1f2efdc2-5d11-4ab3-959c-a6db5dddc331
80702025-04-30T06:26:37.596ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35580,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54662 req_id = 1f2efdc2-5d11-4ab3-959c-a6db5dddc331
80712025-04-30T06:26:37.596ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
80722025-04-30T06:26:37.596ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54662 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1f2efdc2-5d11-4ab3-959c-a6db5dddc331
80732025-04-30T06:26:37.663ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
80742025-04-30T06:26:37.664ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80752025-04-30T06:26:37.664ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80762025-04-30T06:26:37.664ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80772025-04-30T06:26:37.664ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
80782025-04-30T06:26:37.665ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80792025-04-30T06:26:37.666ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ee a0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53963 req_id = eb3746ff-be96-4315-9b77-1fb71105c8a9
80802025-04-30T06:26:37.666ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61088,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53963 req_id = eb3746ff-be96-4315-9b77-1fb71105c8a9
80812025-04-30T06:26:37.666ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
80822025-04-30T06:26:37.666ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61088, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53963 req_id = eb3746ff-be96-4315-9b77-1fb71105c8a9
80832025-04-30T06:26:37.667ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
80842025-04-30T06:26:37.667ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
80852025-04-30T06:26:37.667ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
80862025-04-30T06:26:37.668ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80872025-04-30T06:26:37.678ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80882025-04-30T06:26:37.678ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80892025-04-30T06:26:37.678ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
80902025-04-30T06:26:37.679ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80912025-04-30T06:26:37.682ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c1e2a6e-cb20-4d95-b19d-96827045c51d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
80922025-04-30T06:26:37.682ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
80932025-04-30T06:26:37.683ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80942025-04-30T06:26:37.683ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
80952025-04-30T06:26:37.683ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80962025-04-30T06:26:37.685ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b32a3db5-f136-43c3-8eb5-841ba63ef308", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
80972025-04-30T06:26:37.685ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
80982025-04-30T06:26:37.689ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8030ffc-62b3-4711-910c-8f09e4733788", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
80992025-04-30T06:26:37.689ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
81002025-04-30T06:26:37.692ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "340eb94f-0474-4976-9bda-7b520d8a2e26", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81012025-04-30T06:26:37.692ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
81022025-04-30T06:26:37.702ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb0a6bad-bc18-404a-b553-3b639954d061", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81032025-04-30T06:26:37.703ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
81042025-04-30T06:26:37.705ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77f9b5c5-dad5-48b6-b508-052b75697eb8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81052025-04-30T06:26:37.705ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
81062025-04-30T06:26:37.709ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e76a68f7-4764-4258-915f-74f5a0bc7f0b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81072025-04-30T06:26:37.709ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
81082025-04-30T06:26:37.713ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "16e967d0-b21d-42d7-98cb-1dc43c1e0c69", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81092025-04-30T06:26:37.714ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
81102025-04-30T06:26:37.716ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b127a047-c59a-4e2e-8217-d466471003ac", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81112025-04-30T06:26:37.716ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
81122025-04-30T06:26:37.720ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8ea1404-40a3-4112-8ccb-1442343523dd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81132025-04-30T06:26:37.720ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
81142025-04-30T06:26:37.721ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c75030bf-fa6f-460a-9597-82f06d3cfe8f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81152025-04-30T06:26:37.721ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
81162025-04-30T06:26:37.724ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6656f35c-f4ad-4771-9c6d-bf3a093c5c5c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81172025-04-30T06:26:37.724ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
81182025-04-30T06:26:37.729ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
81192025-04-30T06:26:37.737ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e98fae77-06f6-4155-bef4-5fa945b6d1e1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81202025-04-30T06:26:37.737ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
81212025-04-30T06:26:37.744ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f69ad38-16a3-45ee-8527-dcb4197ee9e7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81222025-04-30T06:26:37.744ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
81232025-04-30T06:26:37.747ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "856d7194-027f-4432-adf4-3e80b0c177aa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81242025-04-30T06:26:37.747ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
81252025-04-30T06:26:37.751ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dada4139-5d11-42b3-b2fa-5602bc5f132d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81262025-04-30T06:26:37.751ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
81272025-04-30T06:26:37.754ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7281b61f-2b93-4a28-a233-b816a3753fb8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81282025-04-30T06:26:37.755ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
81292025-04-30T06:26:37.758ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28ee24de-2634-42f0-b0a0-1217e68857ff", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81302025-04-30T06:26:37.758ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
81312025-04-30T06:26:37.762ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63afc3ef-06b2-4338-a671-2270440bb330", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81322025-04-30T06:26:37.762ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
81332025-04-30T06:26:37.765ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca52aefd-79e5-4f37-92f3-450377337cbe", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81342025-04-30T06:26:37.765ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
81352025-04-30T06:26:37.766ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b71a18ee-d9ad-49e3-8192-4f807f74f223", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81362025-04-30T06:26:37.766ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
81372025-04-30T06:26:37.768ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed26c87a-5dc3-46ae-92e0-688e45103a14", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81382025-04-30T06:26:37.769ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
81392025-04-30T06:26:37.770ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4573fd1f-b907-4533-92f7-35aa0747a780", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81402025-04-30T06:26:37.770ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
81412025-04-30T06:26:37.772ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6257fee7-b9c9-448a-af12-4f7cf4a553be", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81422025-04-30T06:26:37.772ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
81432025-04-30T06:26:37.776ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbcae290-5080-4601-9ef1-5412a2199455", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81442025-04-30T06:26:37.776ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
81452025-04-30T06:26:37.779ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c42ad239-c0bb-44ae-ad21-e72dcaa9adb1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81462025-04-30T06:26:37.779ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
81472025-04-30T06:26:37.780ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81482025-04-30T06:26:37.780ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81492025-04-30T06:26:37.780ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81502025-04-30T06:26:37.780ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
81512025-04-30T06:26:37.782ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81522025-04-30T06:26:37.786ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e427dee-48ed-479e-a97d-68d53aee4dda", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81532025-04-30T06:26:37.787ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
81542025-04-30T06:26:37.794ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81552025-04-30T06:26:37.807ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81562025-04-30T06:26:37.807ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81572025-04-30T06:26:37.807ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
81582025-04-30T06:26:37.808ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81592025-04-30T06:26:37.812ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81602025-04-30T06:26:37.812ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
81612025-04-30T06:26:37.813ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81622025-04-30T06:26:37.816ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "162ca796-b48e-4f4b-8ef4-33e34f78137a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81632025-04-30T06:26:37.816ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
81642025-04-30T06:26:37.827ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dfdaabf-415b-446f-954a-e6f409f5ad07", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81652025-04-30T06:26:37.827ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
81662025-04-30T06:26:37.832ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae5b583d-196d-40b7-b62a-3e12545734f1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81672025-04-30T06:26:37.833ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
81682025-04-30T06:26:37.835ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56323922-7d91-4e5d-9ec7-294b6c52ef6d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81692025-04-30T06:26:37.835ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
81702025-04-30T06:26:37.838ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44aba7a2-3429-4e4b-8d45-9d33319ca87a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81712025-04-30T06:26:37.838ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
81722025-04-30T06:26:37.845ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4633df6c-9a79-4ab8-9753-2c64c6a85226", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81732025-04-30T06:26:37.845ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
81742025-04-30T06:26:37.847ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7baac81-626d-4ec7-9913-d312233ed390", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81752025-04-30T06:26:37.848ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
81762025-04-30T06:26:37.852ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
81772025-04-30T06:26:37.852ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1124aee-6e7b-427d-b607-7cd833f0dc56", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81782025-04-30T06:26:37.853ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
81792025-04-30T06:26:37.859ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bba1dfaa-77e6-422d-87a7-dded4e1ce6cc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81802025-04-30T06:26:37.860ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
81812025-04-30T06:26:37.862ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c03cd3ad-b8c7-481f-846b-725ce26ac970", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81822025-04-30T06:26:37.862ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
81832025-04-30T06:26:37.865ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8a31ff8-6bcb-4442-a2fc-8780bb2182ee", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81842025-04-30T06:26:37.865ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
81852025-04-30T06:26:37.868ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "345e97fa-1676-45b3-8e31-1309c2612125", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81862025-04-30T06:26:37.868ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
81872025-04-30T06:26:37.871ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93d1f9bc-4dc5-426e-a315-8fe606739ad3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81882025-04-30T06:26:37.873ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
81892025-04-30T06:26:37.889ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36458cbd-3988-45fb-845b-42f4e701ba65", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81902025-04-30T06:26:37.889ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
81912025-04-30T06:26:37.890ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca4838a7-4635-45ed-ba90-fed2963ad9c7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81922025-04-30T06:26:37.890ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
81932025-04-30T06:26:37.892ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a126631f-2d45-4c2d-9763-2813c37d2028", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81942025-04-30T06:26:37.892ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
81952025-04-30T06:26:37.893ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fd2df9b-9937-4083-888f-a065697cdf34", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81962025-04-30T06:26:37.893ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
81972025-04-30T06:26:37.894ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ffe73e1-8a60-4e47-b29c-313168465bef", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
81982025-04-30T06:26:37.894ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
81992025-04-30T06:26:37.894ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82002025-04-30T06:26:37.894ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82012025-04-30T06:26:37.894ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82022025-04-30T06:26:37.895ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82032025-04-30T06:26:37.900ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82042025-04-30T06:26:37.907ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
82052025-04-30T06:26:37.907ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
82062025-04-30T06:26:37.907ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82072025-04-30T06:26:37.907ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82082025-04-30T06:26:37.907ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82092025-04-30T06:26:37.908ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82102025-04-30T06:26:37.913ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82112025-04-30T06:26:37.913ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82122025-04-30T06:26:37.913ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82132025-04-30T06:26:37.913ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
82142025-04-30T06:26:37.914ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82152025-04-30T06:26:37.916ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82162025-04-30T06:26:37.920ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
82172025-04-30T06:26:37.921ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82182025-04-30T06:26:37.921ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82192025-04-30T06:26:37.921ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82202025-04-30T06:26:37.923ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82212025-04-30T06:26:37.930ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82222025-04-30T06:26:37.930ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82232025-04-30T06:26:37.930ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
82242025-04-30T06:26:37.931ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82252025-04-30T06:26:38.004ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
82262025-04-30T06:26:38.004ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
82272025-04-30T06:26:38.005ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:56323 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
82282025-04-30T06:26:38.006ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
82292025-04-30T06:26:38.006ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
82302025-04-30T06:26:38.007ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
82312025-04-30T06:26:38.009ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
82322025-04-30T06:26:38.009ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82332025-04-30T06:26:38.009ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
82342025-04-30T06:26:38.010ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82352025-04-30T06:26:38.010ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:56323 req_id = 44fd85ab-4703-4ede-a5c7-2b9185855ec7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
82362025-04-30T06:26:38.011ZINFOtest_disk_expunge (dropshot): request completed latency_us = 805 local_addr = [::1]:44559 method = GET remote_addr = [::1]:56323 req_id = 44fd85ab-4703-4ede-a5c7-2b9185855ec7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
82372025-04-30T06:26:38.012ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "44fd85ab-4703-4ede-a5c7-2b9185855ec7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
82382025-04-30T06:26:38.017ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
82392025-04-30T06:26:38.018ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
82402025-04-30T06:26:38.018ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
82412025-04-30T06:26:38.021ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
82422025-04-30T06:26:38.021ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82432025-04-30T06:26:38.021ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82442025-04-30T06:26:38.021ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82452025-04-30T06:26:38.022ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82462025-04-30T06:26:38.023ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
82472025-04-30T06:26:38.023ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
82482025-04-30T06:26:38.028ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82492025-04-30T06:26:38.035ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
82502025-04-30T06:26:38.036ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82512025-04-30T06:26:38.036ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82522025-04-30T06:26:38.036ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82532025-04-30T06:26:38.038ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
82542025-04-30T06:26:38.038ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
82552025-04-30T06:26:38.045ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82562025-04-30T06:26:38.045ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82572025-04-30T06:26:38.052ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82582025-04-30T06:26:38.053ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82592025-04-30T06:26:38.053ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82602025-04-30T06:26:38.053ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82612025-04-30T06:26:38.113ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
82622025-04-30T06:26:38.115ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
82632025-04-30T06:26:38.117ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
82642025-04-30T06:26:38.117ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
82652025-04-30T06:26:38.119ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 542.625095ms iteration = 26 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
82662025-04-30T06:26:38.119ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 27 reason = Signaled
82672025-04-30T06:26:38.119ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
82682025-04-30T06:26:38.119ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82692025-04-30T06:26:38.119ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82702025-04-30T06:26:38.119ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82712025-04-30T06:26:38.120ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82722025-04-30T06:26:38.132ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82732025-04-30T06:26:38.142ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
82742025-04-30T06:26:38.142ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82752025-04-30T06:26:38.142ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82762025-04-30T06:26:38.143ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82772025-04-30T06:26:38.148ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
82782025-04-30T06:26:38.149ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
82792025-04-30T06:26:38.152ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a1 d1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60619 req_id = 2d23185c-f213-4e00-bccf-b4a471d7ef88
82802025-04-30T06:26:38.153ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41425,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60619 req_id = 2d23185c-f213-4e00-bccf-b4a471d7ef88
82812025-04-30T06:26:38.153ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
82822025-04-30T06:26:38.153ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82832025-04-30T06:26:38.153ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60619 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2d23185c-f213-4e00-bccf-b4a471d7ef88
82842025-04-30T06:26:38.154ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3262, tv_nsec: 953705757 } })
82852025-04-30T06:26:38.155ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
82862025-04-30T06:26:38.155ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 54 d4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e T...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50000 req_id = 3c6cfd0b-43af-4cb4-b10f-d84be1c3aea7
82872025-04-30T06:26:38.155ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21716,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50000 req_id = 3c6cfd0b-43af-4cb4-b10f-d84be1c3aea7
82882025-04-30T06:26:38.155ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
82892025-04-30T06:26:38.155ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50000 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3c6cfd0b-43af-4cb4-b10f-d84be1c3aea7
82902025-04-30T06:26:38.241ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
82912025-04-30T06:26:38.241ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82922025-04-30T06:26:38.242ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82932025-04-30T06:26:38.242ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82942025-04-30T06:26:38.242ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
82952025-04-30T06:26:38.242ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82962025-04-30T06:26:38.243ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 80 5c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .\\..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52222 req_id = a0de4d6c-4e48-4c0d-a8ab-f3d1493f316a
82972025-04-30T06:26:38.244ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32860,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52222 req_id = a0de4d6c-4e48-4c0d-a8ab-f3d1493f316a
82982025-04-30T06:26:38.244ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
82992025-04-30T06:26:38.244ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32860, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52222 req_id = a0de4d6c-4e48-4c0d-a8ab-f3d1493f316a
83002025-04-30T06:26:38.244ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
83012025-04-30T06:26:38.244ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
83022025-04-30T06:26:38.244ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
83032025-04-30T06:26:38.246ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83042025-04-30T06:26:38.252ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83052025-04-30T06:26:38.252ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83062025-04-30T06:26:38.252ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
83072025-04-30T06:26:38.253ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83082025-04-30T06:26:38.253ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e76d7c33-d3b5-401f-8fb7-2785e3fc3278", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83092025-04-30T06:26:38.253ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
83102025-04-30T06:26:38.255ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b73cc91d-dec6-4658-98de-8d05ff4ce2cf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83112025-04-30T06:26:38.256ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
83122025-04-30T06:26:38.256ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83132025-04-30T06:26:38.257ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
83142025-04-30T06:26:38.258ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83152025-04-30T06:26:38.263ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29369b1d-2133-43c3-85b8-81ec0e015e1a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83162025-04-30T06:26:38.263ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
83172025-04-30T06:26:38.267ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6715ac1c-7978-49c8-bc01-13ecad31edf9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83182025-04-30T06:26:38.268ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
83192025-04-30T06:26:38.272ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83949bb2-1b8e-4e7c-bc52-7b7005fa652d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83202025-04-30T06:26:38.272ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
83212025-04-30T06:26:38.277ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97dece9a-f43f-4c7a-9ec5-8c0460ea5b3c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83222025-04-30T06:26:38.278ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
83232025-04-30T06:26:38.282ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "615c9b8e-f64d-4214-974a-065760021a05", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83242025-04-30T06:26:38.282ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
83252025-04-30T06:26:38.291ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ecbe05b-1c9d-4f24-9d9e-090a26e04084", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83262025-04-30T06:26:38.291ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
83272025-04-30T06:26:38.292ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
83282025-04-30T06:26:38.295ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5264be0-d038-42d5-9f9c-69ad9a1658e2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83292025-04-30T06:26:38.295ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
83302025-04-30T06:26:38.300ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a2c367c-33d8-4e4b-912a-52c7f9356764", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83312025-04-30T06:26:38.300ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
83322025-04-30T06:26:38.301ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a657213b-174f-45a7-865f-727330019487", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83332025-04-30T06:26:38.301ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
83342025-04-30T06:26:38.303ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee3a7992-5631-497d-9df1-57c983759d9c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83352025-04-30T06:26:38.303ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
83362025-04-30T06:26:38.304ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a9520f8-4cad-48cd-9592-d8af0a787ee1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83372025-04-30T06:26:38.305ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
83382025-04-30T06:26:38.307ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "26522165-c119-4967-bdd4-ce110fab5ea4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83392025-04-30T06:26:38.307ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
83402025-04-30T06:26:38.309ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f55c543-c0d9-4d82-83c7-3f0117d01a0f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83412025-04-30T06:26:38.309ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
83422025-04-30T06:26:38.311ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "062c916d-9ca4-4561-89f2-8753b7eec4a3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83432025-04-30T06:26:38.311ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
83442025-04-30T06:26:38.313ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e1f001e-c30d-4284-80ac-33986863cd48", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83452025-04-30T06:26:38.313ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
83462025-04-30T06:26:38.315ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ab6a565-de22-4a8a-b44e-c552cee0e48c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83472025-04-30T06:26:38.315ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
83482025-04-30T06:26:38.317ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cada593e-8744-45fc-94ec-98a55a10dfa3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83492025-04-30T06:26:38.317ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
83502025-04-30T06:26:38.319ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba85474e-935f-459d-af2b-2a2cdfd2e80c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83512025-04-30T06:26:38.319ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
83522025-04-30T06:26:38.321ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d8c5b25-e9b4-4c77-9350-869ed716e25a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83532025-04-30T06:26:38.321ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
83542025-04-30T06:26:38.324ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1a97626-7b3a-4084-bf2a-8ff590cc8c21", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83552025-04-30T06:26:38.324ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
83562025-04-30T06:26:38.326ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f001643a-717c-4cff-8819-8a64a2a9ca6a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83572025-04-30T06:26:38.326ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
83582025-04-30T06:26:38.328ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9296e5e5-4a57-4230-87eb-cb1594773fa4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:37 GMT"} })
83592025-04-30T06:26:38.328ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
83602025-04-30T06:26:38.331ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f23505f-0e51-4eda-a0ae-248fb2f533f3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83612025-04-30T06:26:38.331ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
83622025-04-30T06:26:38.333ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f332994-a287-477f-9722-5ad24dc01e5a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83632025-04-30T06:26:38.333ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
83642025-04-30T06:26:38.335ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c9b7245-6c7a-44a4-9201-a35f741f755b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83652025-04-30T06:26:38.335ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
83662025-04-30T06:26:38.337ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ccd562b-1999-44ae-b337-6c00c5840e28", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83672025-04-30T06:26:38.337ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
83682025-04-30T06:26:38.339ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3609b252-bfd7-4b33-93c5-e9e8831b769c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83692025-04-30T06:26:38.340ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
83702025-04-30T06:26:38.342ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb3b6450-2c85-4b09-8474-62e93b7cc330", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83712025-04-30T06:26:38.342ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
83722025-04-30T06:26:38.343ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83732025-04-30T06:26:38.343ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83742025-04-30T06:26:38.343ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83752025-04-30T06:26:38.343ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
83762025-04-30T06:26:38.346ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83772025-04-30T06:26:38.347ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f114e67e-25e5-409a-b425-c5e6b3741af3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83782025-04-30T06:26:38.347ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
83792025-04-30T06:26:38.347ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77cf2316-7f7b-4f07-a3a4-030356443727", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83802025-04-30T06:26:38.348ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
83812025-04-30T06:26:38.348ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83822025-04-30T06:26:38.355ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83832025-04-30T06:26:38.355ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83842025-04-30T06:26:38.355ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
83852025-04-30T06:26:38.355ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83862025-04-30T06:26:38.358ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eeb95d3e-c6d3-4e97-ae36-693208c9d31a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83872025-04-30T06:26:38.358ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
83882025-04-30T06:26:38.358ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83892025-04-30T06:26:38.358ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
83902025-04-30T06:26:38.359ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83912025-04-30T06:26:38.360ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ea4a244-179c-4f8b-a6b0-d70e8c621a07", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83922025-04-30T06:26:38.361ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
83932025-04-30T06:26:38.363ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7303253c-b12f-4a1d-937a-f5bf3f640be2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83942025-04-30T06:26:38.363ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
83952025-04-30T06:26:38.368ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30c5c371-2c29-4e44-9122-76cea887edf4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83962025-04-30T06:26:38.369ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
83972025-04-30T06:26:38.371ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88d76de1-782a-43c7-a0b4-01dcf0bb9587", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
83982025-04-30T06:26:38.371ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
83992025-04-30T06:26:38.375ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e815e78d-b94e-4d0d-aac1-644ff15e2d08", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
84002025-04-30T06:26:38.375ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
84012025-04-30T06:26:38.380ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21e604b5-277e-4dab-93ea-83e80521eed9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
84022025-04-30T06:26:38.380ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
84032025-04-30T06:26:38.383ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0b7810e-dbdc-4571-aa8b-e7b3b29c6b1f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
84042025-04-30T06:26:38.383ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
84052025-04-30T06:26:38.387ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a99c778f-ff05-480d-8eab-88fc559a1c82", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
84062025-04-30T06:26:38.387ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
84072025-04-30T06:26:38.387ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
84082025-04-30T06:26:38.390ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bcbd903-b91f-48ab-ad28-c875de48a854", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
84092025-04-30T06:26:38.390ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
84102025-04-30T06:26:38.391ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0584f5c4-4780-4f6e-a3cf-680f54ffc945", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
84112025-04-30T06:26:38.391ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
84122025-04-30T06:26:38.393ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fa8526c-09e5-4753-bbe4-756070a40037", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
84132025-04-30T06:26:38.393ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
84142025-04-30T06:26:38.394ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0872933f-bdff-4b03-a272-9ca625ac57fc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
84152025-04-30T06:26:38.394ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
84162025-04-30T06:26:38.394ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84172025-04-30T06:26:38.394ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84182025-04-30T06:26:38.394ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84192025-04-30T06:26:38.394ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84202025-04-30T06:26:38.396ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84212025-04-30T06:26:38.406ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
84222025-04-30T06:26:38.406ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
84232025-04-30T06:26:38.406ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84242025-04-30T06:26:38.406ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84252025-04-30T06:26:38.406ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84262025-04-30T06:26:38.407ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84272025-04-30T06:26:38.411ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84282025-04-30T06:26:38.420ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
84292025-04-30T06:26:38.421ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84302025-04-30T06:26:38.421ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84312025-04-30T06:26:38.421ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84322025-04-30T06:26:38.494ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
84332025-04-30T06:26:38.494ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
84342025-04-30T06:26:38.495ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84352025-04-30T06:26:38.495ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84362025-04-30T06:26:38.495ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84372025-04-30T06:26:38.495ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
84382025-04-30T06:26:38.496ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84392025-04-30T06:26:38.497ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:60899 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
84402025-04-30T06:26:38.498ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:60899 req_id = 721e08b8-3659-4f0b-8f88-d127c7fa8e7a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
84412025-04-30T06:26:38.499ZINFOtest_disk_expunge (dropshot): request completed latency_us = 479 local_addr = [::1]:44559 method = GET remote_addr = [::1]:60899 req_id = 721e08b8-3659-4f0b-8f88-d127c7fa8e7a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
84422025-04-30T06:26:38.500ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "721e08b8-3659-4f0b-8f88-d127c7fa8e7a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
84432025-04-30T06:26:38.502ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
84442025-04-30T06:26:38.502ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
84452025-04-30T06:26:38.502ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
84462025-04-30T06:26:38.506ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
84472025-04-30T06:26:38.506ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84482025-04-30T06:26:38.506ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84492025-04-30T06:26:38.506ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84502025-04-30T06:26:38.507ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84512025-04-30T06:26:38.508ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84522025-04-30T06:26:38.515ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84532025-04-30T06:26:38.515ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84542025-04-30T06:26:38.516ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
84552025-04-30T06:26:38.516ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84562025-04-30T06:26:38.518ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84572025-04-30T06:26:38.534ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84582025-04-30T06:26:38.534ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84592025-04-30T06:26:38.534ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84602025-04-30T06:26:38.535ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84612025-04-30T06:26:38.535ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84622025-04-30T06:26:38.535ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
84632025-04-30T06:26:38.535ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84642025-04-30T06:26:38.572ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
84652025-04-30T06:26:38.573ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
84662025-04-30T06:26:38.573ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 454.160486ms iteration = 27 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
84672025-04-30T06:26:38.573ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 28 reason = Signaled
84682025-04-30T06:26:38.573ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
84692025-04-30T06:26:38.573ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84702025-04-30T06:26:38.573ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84712025-04-30T06:26:38.573ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84722025-04-30T06:26:38.574ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84732025-04-30T06:26:38.583ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
84742025-04-30T06:26:38.584ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84752025-04-30T06:26:38.590ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84762025-04-30T06:26:38.590ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84772025-04-30T06:26:38.590ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84782025-04-30T06:26:38.593ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84792025-04-30T06:26:38.596ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
84802025-04-30T06:26:38.596ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
84812025-04-30T06:26:38.597ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: dc 70 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .p..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51865 req_id = b90f6e48-4488-4de4-8256-4e1dd97c213f
84822025-04-30T06:26:38.597ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56432,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51865 req_id = b90f6e48-4488-4de4-8256-4e1dd97c213f
84832025-04-30T06:26:38.597ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
84842025-04-30T06:26:38.597ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84852025-04-30T06:26:38.598ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51865 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b90f6e48-4488-4de4-8256-4e1dd97c213f
84862025-04-30T06:26:38.598ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3263, tv_nsec: 397287004 } })
84872025-04-30T06:26:38.598ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
84882025-04-30T06:26:38.599ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: aa e8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49589 req_id = 87e0297d-6755-4df1-8f07-ad4f0e8b8eef
84892025-04-30T06:26:38.599ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43752,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49589 req_id = 87e0297d-6755-4df1-8f07-ad4f0e8b8eef
84902025-04-30T06:26:38.599ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84912025-04-30T06:26:38.599ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49589 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 87e0297d-6755-4df1-8f07-ad4f0e8b8eef
84922025-04-30T06:26:38.660ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
84932025-04-30T06:26:38.662ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84942025-04-30T06:26:38.662ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84952025-04-30T06:26:38.662ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84962025-04-30T06:26:38.662ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
84972025-04-30T06:26:38.662ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84982025-04-30T06:26:38.663ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: cc 5e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .^..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65014 req_id = 1005d7ac-03b6-4488-8091-fa12956fe8c9
84992025-04-30T06:26:38.663ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52318,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65014 req_id = 1005d7ac-03b6-4488-8091-fa12956fe8c9
85002025-04-30T06:26:38.664ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
85012025-04-30T06:26:38.664ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52318, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65014 req_id = 1005d7ac-03b6-4488-8091-fa12956fe8c9
85022025-04-30T06:26:38.664ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
85032025-04-30T06:26:38.664ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
85042025-04-30T06:26:38.664ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
85052025-04-30T06:26:38.667ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85062025-04-30T06:26:38.681ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85072025-04-30T06:26:38.681ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85082025-04-30T06:26:38.681ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
85092025-04-30T06:26:38.682ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85102025-04-30T06:26:38.685ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c68b28ba-53e4-4f60-a604-66019bb69840", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85112025-04-30T06:26:38.685ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
85122025-04-30T06:26:38.687ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85132025-04-30T06:26:38.687ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
85142025-04-30T06:26:38.689ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85152025-04-30T06:26:38.690ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e34899bf-d9fc-45d8-8ccd-3cd14170de72", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85162025-04-30T06:26:38.690ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
85172025-04-30T06:26:38.694ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5b57329-892c-45e2-8e9d-9c249da34cae", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85182025-04-30T06:26:38.694ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
85192025-04-30T06:26:38.697ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ced5ba4-4a5b-47cd-ba8d-8b317c92b37f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85202025-04-30T06:26:38.698ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
85212025-04-30T06:26:38.700ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eb0251c-12e4-435c-b2ec-5f26bffc9953", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85222025-04-30T06:26:38.700ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
85232025-04-30T06:26:38.703ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70a6b6ad-a28b-4e78-a4c7-bdd58ceba223", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85242025-04-30T06:26:38.703ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
85252025-04-30T06:26:38.706ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "53eb1804-a618-4ad2-87d8-935aafc48c22", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85262025-04-30T06:26:38.707ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
85272025-04-30T06:26:38.711ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff0e8da2-b48a-4c56-840d-ba8c3c172841", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85282025-04-30T06:26:38.712ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
85292025-04-30T06:26:38.714ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15aa8437-c945-4ca9-a141-2d579960d7a2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85302025-04-30T06:26:38.714ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
85312025-04-30T06:26:38.715ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dae687a-abcd-444a-aa74-e6d8e2470cb8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85322025-04-30T06:26:38.717ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
85332025-04-30T06:26:38.732ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1224f22-8bb7-4b9c-a7ca-9823d47e5d61", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85342025-04-30T06:26:38.732ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
85352025-04-30T06:26:38.734ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a42bb63-5822-4525-809b-2508eb640cb9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85362025-04-30T06:26:38.734ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
85372025-04-30T06:26:38.737ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af89dbd1-98f2-47f2-87f7-d9b685129566", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85382025-04-30T06:26:38.737ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
85392025-04-30T06:26:38.737ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
85402025-04-30T06:26:38.740ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79d0e7f1-53d5-4778-bc46-804daf376f95", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85412025-04-30T06:26:38.740ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
85422025-04-30T06:26:38.743ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b60f4268-c47e-4933-9906-119233961e2f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85432025-04-30T06:26:38.743ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
85442025-04-30T06:26:38.745ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b74e70d-4afa-4dc4-8062-d51821e4ff3f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85452025-04-30T06:26:38.746ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
85462025-04-30T06:26:38.749ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1f594ea-d13c-45fc-b762-096d985f11bc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85472025-04-30T06:26:38.750ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
85482025-04-30T06:26:38.753ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05714f2f-8fba-4a2c-865b-ca8fa65921d0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85492025-04-30T06:26:38.753ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
85502025-04-30T06:26:38.755ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ca103ca-3b53-4dea-aa36-928525b9dbca", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85512025-04-30T06:26:38.756ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
85522025-04-30T06:26:38.757ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07bbb1a8-4a07-4e17-80e7-db2e6a7afe14", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85532025-04-30T06:26:38.757ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
85542025-04-30T06:26:38.759ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f48a7c77-8298-40bf-be63-38effd2f09db", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85552025-04-30T06:26:38.759ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
85562025-04-30T06:26:38.761ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21c6ead7-6b78-42b9-85eb-c9b56f16028f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85572025-04-30T06:26:38.761ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
85582025-04-30T06:26:38.762ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95cfe3ae-bd8e-4b6d-a965-3b742d1e49e2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85592025-04-30T06:26:38.763ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
85602025-04-30T06:26:38.764ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21e674a9-9db1-495f-a625-dc30a9f4afd6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85612025-04-30T06:26:38.764ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
85622025-04-30T06:26:38.769ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "200b281e-3c8d-4c10-bcf3-15e84198ced1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85632025-04-30T06:26:38.769ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
85642025-04-30T06:26:38.774ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "917af9a2-a986-4651-8715-db4de11d1fb7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85652025-04-30T06:26:38.774ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
85662025-04-30T06:26:38.778ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e83e1806-f74e-4080-bfff-59da658df911", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85672025-04-30T06:26:38.779ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
85682025-04-30T06:26:38.791ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85692025-04-30T06:26:38.791ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85702025-04-30T06:26:38.791ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85712025-04-30T06:26:38.791ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
85722025-04-30T06:26:38.792ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85732025-04-30T06:26:38.794ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4f713e4-13d4-4ed6-9b41-6cde7ac88c22", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85742025-04-30T06:26:38.794ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
85752025-04-30T06:26:38.794ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85762025-04-30T06:26:38.802ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85772025-04-30T06:26:38.802ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85782025-04-30T06:26:38.802ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
85792025-04-30T06:26:38.802ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85802025-04-30T06:26:38.805ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d807d9d-ed30-43d3-821d-4c8a7dd89024", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85812025-04-30T06:26:38.805ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
85822025-04-30T06:26:38.805ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85832025-04-30T06:26:38.805ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
85842025-04-30T06:26:38.806ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85852025-04-30T06:26:38.808ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40cfcb04-4652-48cf-8213-fb24a7ab3513", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85862025-04-30T06:26:38.808ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
85872025-04-30T06:26:38.809ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e00c3966-5b9e-4ba7-be21-a40c96cd9f04", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85882025-04-30T06:26:38.809ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
85892025-04-30T06:26:38.810ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84066c15-529d-411b-9aa9-7e5d918a93d5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85902025-04-30T06:26:38.811ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
85912025-04-30T06:26:38.812ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a68ea47b-30ec-4307-8ffe-cb60da843e6d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85922025-04-30T06:26:38.812ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
85932025-04-30T06:26:38.813ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7b0db2c-f915-4987-af8d-57fb115df838", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85942025-04-30T06:26:38.813ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
85952025-04-30T06:26:38.816ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9264497-9717-438e-a6e6-a57adc0e85eb", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85962025-04-30T06:26:38.816ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
85972025-04-30T06:26:38.823ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3c3affc-3ca6-4cda-9b57-c72defd5297d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
85982025-04-30T06:26:38.823ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
85992025-04-30T06:26:38.827ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cca11602-c6d6-4220-a61d-b3e8961eb008", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
86002025-04-30T06:26:38.828ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
86012025-04-30T06:26:38.832ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7913ebc3-443c-4219-bbca-0a14dd5ae363", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
86022025-04-30T06:26:38.832ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
86032025-04-30T06:26:38.835ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
86042025-04-30T06:26:38.840ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cefe45f9-f997-420a-a95f-45bf58d7a722", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
86052025-04-30T06:26:38.841ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
86062025-04-30T06:26:38.843ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36f44439-496e-43b4-b66a-e27f3cfcd23c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
86072025-04-30T06:26:38.843ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
86082025-04-30T06:26:38.845ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47a26db6-b1f0-4e25-a918-480e79fd1750", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
86092025-04-30T06:26:38.845ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
86102025-04-30T06:26:38.846ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92434d34-de19-4be8-a3f9-e86dd19dba44", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
86112025-04-30T06:26:38.846ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
86122025-04-30T06:26:38.847ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ccbccae-bf21-45a9-8bc8-42143610984b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
86132025-04-30T06:26:38.847ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
86142025-04-30T06:26:38.850ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b05342a2-6ad0-47e7-8e9a-e613a4a19a6d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
86152025-04-30T06:26:38.850ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
86162025-04-30T06:26:38.850ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "840e1fdb-9842-4266-867e-fba6ba322903", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
86172025-04-30T06:26:38.851ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
86182025-04-30T06:26:38.852ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86192025-04-30T06:26:38.852ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86202025-04-30T06:26:38.852ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86212025-04-30T06:26:38.852ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86222025-04-30T06:26:38.853ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86232025-04-30T06:26:38.858ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
86242025-04-30T06:26:38.858ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
86252025-04-30T06:26:38.858ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86262025-04-30T06:26:38.858ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86272025-04-30T06:26:38.858ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86282025-04-30T06:26:38.859ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86292025-04-30T06:26:38.863ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86302025-04-30T06:26:38.868ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
86312025-04-30T06:26:38.868ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86322025-04-30T06:26:38.868ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86332025-04-30T06:26:38.869ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86342025-04-30T06:26:38.938ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
86352025-04-30T06:26:38.938ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
86362025-04-30T06:26:38.939ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86372025-04-30T06:26:38.939ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86382025-04-30T06:26:38.939ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86392025-04-30T06:26:38.939ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
86402025-04-30T06:26:38.940ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86412025-04-30T06:26:38.940ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:58985 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
86422025-04-30T06:26:38.940ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
86432025-04-30T06:26:38.941ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
86442025-04-30T06:26:38.941ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = a0f80b3b-6f6f-4eef-8a1c-f737c1b56827 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
86452025-04-30T06:26:38.941ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:58985 req_id = d480df93-42b3-43d0-94b3-b4167a1e5b26 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86462025-04-30T06:26:38.942ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = a0f80b3b-6f6f-4eef-8a1c-f737c1b56827 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
86472025-04-30T06:26:38.942ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = a0f80b3b-6f6f-4eef-8a1c-f737c1b56827 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
86482025-04-30T06:26:38.942ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = a0f80b3b-6f6f-4eef-8a1c-f737c1b56827 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
86492025-04-30T06:26:38.943ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1209 local_addr = [::1]:44559 method = GET remote_addr = [::1]:58985 req_id = d480df93-42b3-43d0-94b3-b4167a1e5b26 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86502025-04-30T06:26:38.943ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d480df93-42b3-43d0-94b3-b4167a1e5b26", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
86512025-04-30T06:26:38.948ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86522025-04-30T06:26:38.948ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86532025-04-30T06:26:38.948ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
86542025-04-30T06:26:38.955ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86552025-04-30T06:26:38.955ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86562025-04-30T06:26:38.955ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86572025-04-30T06:26:38.955ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86582025-04-30T06:26:38.956ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86592025-04-30T06:26:38.958ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86602025-04-30T06:26:38.966ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86612025-04-30T06:26:38.966ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86622025-04-30T06:26:38.966ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
86632025-04-30T06:26:38.968ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86642025-04-30T06:26:38.968ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86652025-04-30T06:26:38.969ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86662025-04-30T06:26:38.969ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 28008 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = a0f80b3b-6f6f-4eef-8a1c-f737c1b56827 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
86672025-04-30T06:26:38.971ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
86682025-04-30T06:26:38.974ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
86692025-04-30T06:26:38.974ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86702025-04-30T06:26:38.982ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86712025-04-30T06:26:38.984ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86722025-04-30T06:26:38.984ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86732025-04-30T06:26:38.984ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86742025-04-30T06:26:38.986ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0f80b3b-6f6f-4eef-8a1c-f737c1b56827", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
86752025-04-30T06:26:38.986ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
86762025-04-30T06:26:38.986ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d103e79a-01d8-4657-96d3-a1b94c6e89b7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
86772025-04-30T06:26:38.987ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d103e79a-01d8-4657-96d3-a1b94c6e89b7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
86782025-04-30T06:26:38.987ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d103e79a-01d8-4657-96d3-a1b94c6e89b7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
86792025-04-30T06:26:38.988ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d103e79a-01d8-4657-96d3-a1b94c6e89b7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
86802025-04-30T06:26:38.992ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86812025-04-30T06:26:38.992ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
86822025-04-30T06:26:38.992ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86832025-04-30T06:26:38.994ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
86842025-04-30T06:26:38.994ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
86852025-04-30T06:26:38.997ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
86862025-04-30T06:26:38.997ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
86872025-04-30T06:26:39.001ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 14676 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = d103e79a-01d8-4657-96d3-a1b94c6e89b7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
86882025-04-30T06:26:39.004ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "d103e79a-01d8-4657-96d3-a1b94c6e89b7", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
86892025-04-30T06:26:39.004ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
86902025-04-30T06:26:39.004ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86912025-04-30T06:26:39.004ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
86922025-04-30T06:26:39.047ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86932025-04-30T06:26:39.047ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
86942025-04-30T06:26:39.048ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
86952025-04-30T06:26:39.049ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
86962025-04-30T06:26:39.049ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 475.141158ms iteration = 28 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
86972025-04-30T06:26:39.049ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 29 reason = Signaled
86982025-04-30T06:26:39.049ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86992025-04-30T06:26:39.049ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87002025-04-30T06:26:39.049ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87012025-04-30T06:26:39.049ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87022025-04-30T06:26:39.049ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87032025-04-30T06:26:39.059ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87042025-04-30T06:26:39.060ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87052025-04-30T06:26:39.060ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87062025-04-30T06:26:39.060ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87072025-04-30T06:26:39.060ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87082025-04-30T06:26:39.067ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
87092025-04-30T06:26:39.067ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87102025-04-30T06:26:39.067ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87112025-04-30T06:26:39.067ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87122025-04-30T06:26:39.070ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
87132025-04-30T06:26:39.072ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
87142025-04-30T06:26:39.072ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
87152025-04-30T06:26:39.076ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 16 af 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60503 req_id = e0524c11-d538-4883-b0a7-b414f473899a
87162025-04-30T06:26:39.076ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5807,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60503 req_id = e0524c11-d538-4883-b0a7-b414f473899a
87172025-04-30T06:26:39.076ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
87182025-04-30T06:26:39.076ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
87192025-04-30T06:26:39.076ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60503 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e0524c11-d538-4883-b0a7-b414f473899a
87202025-04-30T06:26:39.077ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3263, tv_nsec: 876801656 } })
87212025-04-30T06:26:39.078ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
87222025-04-30T06:26:39.090ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d5 61 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .a..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56169 req_id = 0a3216fa-821b-4218-939b-fa17c4d94edd
87232025-04-30T06:26:39.090ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54625,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56169 req_id = 0a3216fa-821b-4218-939b-fa17c4d94edd
87242025-04-30T06:26:39.090ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
87252025-04-30T06:26:39.090ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56169 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0a3216fa-821b-4218-939b-fa17c4d94edd
87262025-04-30T06:26:39.152ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
87272025-04-30T06:26:39.162ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 61 a0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c a............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63857 req_id = 449be7b7-6c4a-418c-aa07-52216c8387b5
87282025-04-30T06:26:39.163ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24992,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63857 req_id = 449be7b7-6c4a-418c-aa07-52216c8387b5
87292025-04-30T06:26:39.163ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
87302025-04-30T06:26:39.163ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24992, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63857 req_id = 449be7b7-6c4a-418c-aa07-52216c8387b5
87312025-04-30T06:26:39.164ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
87322025-04-30T06:26:39.164ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
87332025-04-30T06:26:39.164ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
87342025-04-30T06:26:39.164ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
87352025-04-30T06:26:39.166ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "505e294d-4195-4c80-8d19-c05f05fed1e2", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87362025-04-30T06:26:39.167ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
87372025-04-30T06:26:39.169ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcfaf46d-d8e3-48ab-ade5-e5de5c9c2d56", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87382025-04-30T06:26:39.169ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
87392025-04-30T06:26:39.171ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1986cc3-09bc-4a4a-9894-8ebe78eb1af9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87402025-04-30T06:26:39.172ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
87412025-04-30T06:26:39.175ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da368f22-2ee6-4ad4-bbe4-c79c9ed1b678", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87422025-04-30T06:26:39.175ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
87432025-04-30T06:26:39.179ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1660de62-dfd5-4244-bdeb-15712e4e8e68", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87442025-04-30T06:26:39.180ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
87452025-04-30T06:26:39.195ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38310a15-ab93-4a87-8282-8f470a903b47", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87462025-04-30T06:26:39.195ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
87472025-04-30T06:26:39.195ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd2e25db-5f80-4b2f-9788-3a35b7adb380", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87482025-04-30T06:26:39.195ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
87492025-04-30T06:26:39.195ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afa7ef81-b197-4f91-9e87-fb5c5d9ce812", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87502025-04-30T06:26:39.195ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
87512025-04-30T06:26:39.195ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f27590af-e65f-4b5f-8429-25f56afdde5e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87522025-04-30T06:26:39.196ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
87532025-04-30T06:26:39.196ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f75cc039-69d9-47bc-a950-9a2208c40a19", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87542025-04-30T06:26:39.196ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
87552025-04-30T06:26:39.196ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fedaf6d-f4f7-4d59-a3d9-7d284f5c546f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87562025-04-30T06:26:39.196ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
87572025-04-30T06:26:39.198ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc6ea00d-0d34-4b89-9130-0d7f8d12721b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87582025-04-30T06:26:39.198ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
87592025-04-30T06:26:39.201ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ea37254-11f7-475d-8472-eab1f23fd009", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87602025-04-30T06:26:39.201ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
87612025-04-30T06:26:39.208ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfe5f801-df9d-4d8d-af7a-aabae2b6712d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87622025-04-30T06:26:39.208ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
87632025-04-30T06:26:39.214ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f79a6da1-d64d-460e-912c-5cdf131faf05", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87642025-04-30T06:26:39.214ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
87652025-04-30T06:26:39.216ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87662025-04-30T06:26:39.216ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87672025-04-30T06:26:39.216ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87682025-04-30T06:26:39.216ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
87692025-04-30T06:26:39.217ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87702025-04-30T06:26:39.219ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7333183c-28cb-48cf-a53d-7c4584aab937", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87712025-04-30T06:26:39.219ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
87722025-04-30T06:26:39.220ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87732025-04-30T06:26:39.228ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87742025-04-30T06:26:39.228ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87752025-04-30T06:26:39.228ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
87762025-04-30T06:26:39.229ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87772025-04-30T06:26:39.231ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "452a30cd-2e18-430d-91d1-6514416fc83c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87782025-04-30T06:26:39.231ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
87792025-04-30T06:26:39.231ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87802025-04-30T06:26:39.231ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
87812025-04-30T06:26:39.232ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87822025-04-30T06:26:39.237ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8953f464-1908-4292-8571-48d80fc2607d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87832025-04-30T06:26:39.238ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
87842025-04-30T06:26:39.240ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b440779-648f-4bbf-8319-76abf6ade2cb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87852025-04-30T06:26:39.240ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
87862025-04-30T06:26:39.241ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b3ecbcc-1248-4d49-a362-46cef3961ce5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87872025-04-30T06:26:39.241ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
87882025-04-30T06:26:39.242ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "091ef54a-6b67-4170-b09c-4714bc459e54", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87892025-04-30T06:26:39.242ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
87902025-04-30T06:26:39.245ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd69c40a-b57f-48ae-87c0-adcb8bf0c71f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87912025-04-30T06:26:39.245ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
87922025-04-30T06:26:39.246ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a7f911a-4776-4ef0-a084-988dd77a7c43", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87932025-04-30T06:26:39.246ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
87942025-04-30T06:26:39.248ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3d43c4b-d878-4408-b8b2-f9885f8291e5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87952025-04-30T06:26:39.248ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
87962025-04-30T06:26:39.250ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67c1444d-df0d-4ed4-89f7-9697b37516c5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87972025-04-30T06:26:39.250ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
87982025-04-30T06:26:39.253ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd1a642f-7fb2-444f-b5c1-10895c764be5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
87992025-04-30T06:26:39.253ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
88002025-04-30T06:26:39.258ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d93f1151-31ad-446b-8df3-a67f62098bbd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88012025-04-30T06:26:39.258ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
88022025-04-30T06:26:39.260ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cbaaaf1-d761-4c86-9748-e01ef0309679", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88032025-04-30T06:26:39.261ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
88042025-04-30T06:26:39.263ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb45b02b-75b7-4572-b586-cbe624c9b1c6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88052025-04-30T06:26:39.263ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
88062025-04-30T06:26:39.266ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
88072025-04-30T06:26:39.269ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6daf55dd-e1f2-440f-b933-b6ed68789c84", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88082025-04-30T06:26:39.269ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
88092025-04-30T06:26:39.270ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23ae9fd7-4781-440c-9e8a-46100e391e7b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88102025-04-30T06:26:39.270ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
88112025-04-30T06:26:39.272ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0ac7333-6ddd-4fd5-b70f-c9688d52696d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88122025-04-30T06:26:39.272ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
88132025-04-30T06:26:39.273ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09fb7b82-2cc0-4f6c-98c9-93b7682ed870", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88142025-04-30T06:26:39.273ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
88152025-04-30T06:26:39.276ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44abe46b-b803-4959-bbcf-2e96c440687b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88162025-04-30T06:26:39.276ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
88172025-04-30T06:26:39.277ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dba3786-27ed-45f3-b1a7-417780696dcc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88182025-04-30T06:26:39.277ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
88192025-04-30T06:26:39.278ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5fd0b98-ff14-4819-8b7f-ebfacb0415ac", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88202025-04-30T06:26:39.279ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
88212025-04-30T06:26:39.281ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af074a76-1657-4bf7-adfa-8478ab8d889b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88222025-04-30T06:26:39.281ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
88232025-04-30T06:26:39.285ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65bdeda2-ae50-4e9c-905b-97e0fe987640", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88242025-04-30T06:26:39.285ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
88252025-04-30T06:26:39.287ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dc84317-60ef-4bd9-8d14-b3d44891c8af", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88262025-04-30T06:26:39.287ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
88272025-04-30T06:26:39.289ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bf4eac8-63ac-4dc7-8358-3803f8499a20", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88282025-04-30T06:26:39.289ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
88292025-04-30T06:26:39.291ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b540c7f-ebc6-40ac-8904-7a98c5e6d29f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88302025-04-30T06:26:39.292ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
88312025-04-30T06:26:39.294ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd68c627-61c0-4c3d-b6c1-10eaadaed4a1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88322025-04-30T06:26:39.294ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
88332025-04-30T06:26:39.295ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "147b31a1-5f83-450b-acbb-b79e8f379d80", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88342025-04-30T06:26:39.296ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
88352025-04-30T06:26:39.297ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f20865c-f5dc-4e08-8f34-c0ef5dfdb47b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88362025-04-30T06:26:39.297ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
88372025-04-30T06:26:39.298ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95191217-7cf1-4f5b-824d-a52439d23ebc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:38 GMT"} })
88382025-04-30T06:26:39.299ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
88392025-04-30T06:26:39.299ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88402025-04-30T06:26:39.299ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88412025-04-30T06:26:39.299ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88422025-04-30T06:26:39.300ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88432025-04-30T06:26:39.303ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88442025-04-30T06:26:39.307ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
88452025-04-30T06:26:39.307ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
88462025-04-30T06:26:39.307ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88472025-04-30T06:26:39.308ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88482025-04-30T06:26:39.308ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88492025-04-30T06:26:39.308ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88502025-04-30T06:26:39.310ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88512025-04-30T06:26:39.314ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
88522025-04-30T06:26:39.314ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88532025-04-30T06:26:39.315ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88542025-04-30T06:26:39.315ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88552025-04-30T06:26:39.317ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88562025-04-30T06:26:39.317ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88572025-04-30T06:26:39.317ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88582025-04-30T06:26:39.317ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
88592025-04-30T06:26:39.319ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88602025-04-30T06:26:39.365ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
88612025-04-30T06:26:39.365ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
88622025-04-30T06:26:39.367ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:59128 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
88632025-04-30T06:26:39.367ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:59128 req_id = d247ad20-cfd4-43df-acbb-c35b6c740b08 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88642025-04-30T06:26:39.368ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1713 local_addr = [::1]:44559 method = GET remote_addr = [::1]:59128 req_id = d247ad20-cfd4-43df-acbb-c35b6c740b08 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
88652025-04-30T06:26:39.368ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88662025-04-30T06:26:39.375ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
88672025-04-30T06:26:39.375ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88682025-04-30T06:26:39.375ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
88692025-04-30T06:26:39.375ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88702025-04-30T06:26:39.376ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d247ad20-cfd4-43df-acbb-c35b6c740b08", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
88712025-04-30T06:26:39.379ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88722025-04-30T06:26:39.380ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
88732025-04-30T06:26:39.380ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
88742025-04-30T06:26:39.382ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88752025-04-30T06:26:39.383ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88762025-04-30T06:26:39.383ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88772025-04-30T06:26:39.383ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88782025-04-30T06:26:39.383ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88792025-04-30T06:26:39.386ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88802025-04-30T06:26:39.386ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
88812025-04-30T06:26:39.387ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88822025-04-30T06:26:39.387ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88832025-04-30T06:26:39.395ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88842025-04-30T06:26:39.396ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88852025-04-30T06:26:39.396ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88862025-04-30T06:26:39.396ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88872025-04-30T06:26:39.438ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
88882025-04-30T06:26:39.438ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
88892025-04-30T06:26:39.438ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
88902025-04-30T06:26:39.438ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 389.97488ms iteration = 29 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
88912025-04-30T06:26:39.438ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 30 reason = Signaled
88922025-04-30T06:26:39.439ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
88932025-04-30T06:26:39.439ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88942025-04-30T06:26:39.439ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
88952025-04-30T06:26:39.439ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
88962025-04-30T06:26:39.440ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
88972025-04-30T06:26:39.442ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88982025-04-30T06:26:39.447ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
88992025-04-30T06:26:39.447ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89002025-04-30T06:26:39.447ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89012025-04-30T06:26:39.447ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89022025-04-30T06:26:39.451ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
89032025-04-30T06:26:39.451ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
89042025-04-30T06:26:39.452ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 2d 36 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 -6..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49974 req_id = 5f5749ea-85ab-4e21-91bd-a19adc1b4929
89052025-04-30T06:26:39.452ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11574,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49974 req_id = 5f5749ea-85ab-4e21-91bd-a19adc1b4929
89062025-04-30T06:26:39.452ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
89072025-04-30T06:26:39.452ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
89082025-04-30T06:26:39.452ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49974 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 5f5749ea-85ab-4e21-91bd-a19adc1b4929
89092025-04-30T06:26:39.452ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3264, tv_nsec: 251853001 } })
89102025-04-30T06:26:39.453ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
89112025-04-30T06:26:39.453ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 65 23 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e e#..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51496 req_id = af95b5b3-e4b0-4731-a4dc-e24cebac529b
89122025-04-30T06:26:39.453ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25891,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51496 req_id = af95b5b3-e4b0-4731-a4dc-e24cebac529b
89132025-04-30T06:26:39.453ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
89142025-04-30T06:26:39.453ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51496 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = af95b5b3-e4b0-4731-a4dc-e24cebac529b
89152025-04-30T06:26:39.498ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
89162025-04-30T06:26:39.499ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89172025-04-30T06:26:39.499ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89182025-04-30T06:26:39.499ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89192025-04-30T06:26:39.499ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
89202025-04-30T06:26:39.500ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89212025-04-30T06:26:39.500ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 88 50 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .P..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53479 req_id = bd0b4000-137d-44ba-aa78-48aa3d4da144
89222025-04-30T06:26:39.500ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34896,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53479 req_id = bd0b4000-137d-44ba-aa78-48aa3d4da144
89232025-04-30T06:26:39.500ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
89242025-04-30T06:26:39.501ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 34896, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53479 req_id = bd0b4000-137d-44ba-aa78-48aa3d4da144
89252025-04-30T06:26:39.501ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
89262025-04-30T06:26:39.501ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
89272025-04-30T06:26:39.501ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
89282025-04-30T06:26:39.502ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89292025-04-30T06:26:39.510ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89302025-04-30T06:26:39.510ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89312025-04-30T06:26:39.510ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
89322025-04-30T06:26:39.511ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89332025-04-30T06:26:39.513ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c4a2d6e-3be1-4d60-87c2-75d2747ce399", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89342025-04-30T06:26:39.513ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
89352025-04-30T06:26:39.514ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89362025-04-30T06:26:39.514ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
89372025-04-30T06:26:39.515ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89382025-04-30T06:26:39.516ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac56ae7d-81a4-493a-b069-304a415a9557", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89392025-04-30T06:26:39.517ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
89402025-04-30T06:26:39.520ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "449be449-bea5-4e69-8f31-8b8c7a232434", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89412025-04-30T06:26:39.520ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
89422025-04-30T06:26:39.523ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e56aecbe-b39a-49a3-ac99-f5e101381065", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89432025-04-30T06:26:39.523ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
89442025-04-30T06:26:39.526ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60fedaa5-1987-4b13-823b-08ef91be29f3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89452025-04-30T06:26:39.526ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
89462025-04-30T06:26:39.530ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "edda925c-33e7-476c-8eaa-c2667140d216", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89472025-04-30T06:26:39.530ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
89482025-04-30T06:26:39.532ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c88d378e-d6b1-40b2-86f2-8ef9000e611d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89492025-04-30T06:26:39.532ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
89502025-04-30T06:26:39.536ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a28661f5-e468-4c93-ba92-47f6a5f7bc19", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89512025-04-30T06:26:39.536ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
89522025-04-30T06:26:39.537ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38680c6d-addf-447f-9ca6-781acc36192e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89532025-04-30T06:26:39.538ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
89542025-04-30T06:26:39.540ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77a34ab7-62ec-4822-8337-cc0d53ada103", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89552025-04-30T06:26:39.541ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
89562025-04-30T06:26:39.542ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f16d87b-2731-4fd9-bbf8-1befdc378e2a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89572025-04-30T06:26:39.542ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
89582025-04-30T06:26:39.545ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9820ed55-6e84-4893-b6b1-c11deb8874ea", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89592025-04-30T06:26:39.545ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
89602025-04-30T06:26:39.545ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
89612025-04-30T06:26:39.547ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07e22b6f-b5a1-4de8-ab9c-1e4dc3cbe719", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89622025-04-30T06:26:39.547ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
89632025-04-30T06:26:39.549ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c34e611-6e68-4038-8027-66b6267be5b9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89642025-04-30T06:26:39.549ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
89652025-04-30T06:26:39.551ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6498bb6-8f0e-47e1-ba0c-ce9f79cca108", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89662025-04-30T06:26:39.551ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
89672025-04-30T06:26:39.554ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac8b9770-c0c3-4700-944a-877c18502682", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89682025-04-30T06:26:39.554ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
89692025-04-30T06:26:39.556ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8754da2c-95ad-4674-901b-55be8c5546e0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89702025-04-30T06:26:39.556ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
89712025-04-30T06:26:39.557ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75b4971f-c278-4ac2-824e-2bedcbf209c5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89722025-04-30T06:26:39.558ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
89732025-04-30T06:26:39.560ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e152fd3-49cd-4974-b6dd-61c0d2e7368b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89742025-04-30T06:26:39.560ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
89752025-04-30T06:26:39.561ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e510429a-bcc1-4934-8111-bed25b6ef75b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89762025-04-30T06:26:39.562ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
89772025-04-30T06:26:39.563ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "388e405f-1d5d-498c-8b7b-4c3542a68e50", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89782025-04-30T06:26:39.563ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
89792025-04-30T06:26:39.565ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb591444-2b6f-42af-9bfc-297b71632ea2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89802025-04-30T06:26:39.565ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
89812025-04-30T06:26:39.566ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a24f4202-7343-41bf-bc9c-0a8eac715bb7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89822025-04-30T06:26:39.567ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
89832025-04-30T06:26:39.569ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff3bed2d-af3b-4498-9687-8e2017f64845", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89842025-04-30T06:26:39.569ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
89852025-04-30T06:26:39.572ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "96a8e64a-33d6-4108-8235-bb3f00f6428b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89862025-04-30T06:26:39.572ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
89872025-04-30T06:26:39.574ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f080c3ba-02d1-49d6-873e-c57106fd5481", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89882025-04-30T06:26:39.575ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
89892025-04-30T06:26:39.577ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3671cee0-4db0-4583-b797-f1509a4e5adf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89902025-04-30T06:26:39.577ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
89912025-04-30T06:26:39.579ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f44b2719-5968-46d7-a749-37e3a9443203", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89922025-04-30T06:26:39.580ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
89932025-04-30T06:26:39.584ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b72d16eb-6e07-49c3-9934-8177d2a71f73", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89942025-04-30T06:26:39.584ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
89952025-04-30T06:26:39.586ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec36ddaf-20a5-4712-9a82-1f5d0be14686", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89962025-04-30T06:26:39.586ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
89972025-04-30T06:26:39.587ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a9508d5-ec42-4c65-aae4-302ed65ee558", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
89982025-04-30T06:26:39.587ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
89992025-04-30T06:26:39.589ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9da8e75e-d6c7-4e1a-97ce-6a2416394aa0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90002025-04-30T06:26:39.589ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
90012025-04-30T06:26:39.590ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5a2c0ca-0e7f-4156-9f18-806704c87a41", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90022025-04-30T06:26:39.590ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
90032025-04-30T06:26:39.591ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd72dcd3-46c3-4f6f-941a-6e3618c7db95", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90042025-04-30T06:26:39.592ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
90052025-04-30T06:26:39.593ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c4e5cda-99c3-418c-9c86-a80290be1654", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90062025-04-30T06:26:39.593ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
90072025-04-30T06:26:39.595ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb9a02e8-f5f2-44ad-b84b-af787bf946fb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90082025-04-30T06:26:39.595ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
90092025-04-30T06:26:39.596ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90102025-04-30T06:26:39.597ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90112025-04-30T06:26:39.597ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90122025-04-30T06:26:39.597ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
90132025-04-30T06:26:39.598ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90142025-04-30T06:26:39.599ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "19bbd079-f773-40b7-8707-fa9c2017fa13", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90152025-04-30T06:26:39.600ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
90162025-04-30T06:26:39.600ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90172025-04-30T06:26:39.606ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90182025-04-30T06:26:39.606ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90192025-04-30T06:26:39.606ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
90202025-04-30T06:26:39.607ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90212025-04-30T06:26:39.610ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90222025-04-30T06:26:39.610ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
90232025-04-30T06:26:39.611ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90242025-04-30T06:26:39.611ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d033410f-16b4-4810-8bb1-7ae2751f386d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90252025-04-30T06:26:39.612ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
90262025-04-30T06:26:39.614ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b879b50-b957-49dc-ae75-0587d148a19f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90272025-04-30T06:26:39.614ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
90282025-04-30T06:26:39.619ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea664853-3d05-4d0e-802c-2c5663de3e90", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90292025-04-30T06:26:39.619ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
90302025-04-30T06:26:39.622ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdac3f2b-a696-41c4-a944-229ef32c242e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90312025-04-30T06:26:39.622ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
90322025-04-30T06:26:39.623ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2d4eb9c-1778-4e29-847a-d9adb6d6ad61", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90332025-04-30T06:26:39.624ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
90342025-04-30T06:26:39.625ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fde59f6a-dd79-4a33-944d-78c570762c19", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90352025-04-30T06:26:39.625ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
90362025-04-30T06:26:39.626ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "185aa528-04b4-48a2-9ece-05e7779cedf1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90372025-04-30T06:26:39.627ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
90382025-04-30T06:26:39.629ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "811451a1-ef2f-4151-8a7b-9f8e02de4f9f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90392025-04-30T06:26:39.629ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90402025-04-30T06:26:39.629ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90412025-04-30T06:26:39.629ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90422025-04-30T06:26:39.629ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90432025-04-30T06:26:39.630ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90442025-04-30T06:26:39.634ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90452025-04-30T06:26:39.646ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90462025-04-30T06:26:39.646ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
90472025-04-30T06:26:39.646ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90482025-04-30T06:26:39.646ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90492025-04-30T06:26:39.646ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90502025-04-30T06:26:39.647ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90512025-04-30T06:26:39.652ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90522025-04-30T06:26:39.659ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
90532025-04-30T06:26:39.659ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90542025-04-30T06:26:39.659ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90552025-04-30T06:26:39.662ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90562025-04-30T06:26:39.758ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
90572025-04-30T06:26:39.758ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
90582025-04-30T06:26:39.760ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:53935 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
90592025-04-30T06:26:39.761ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:53935 req_id = 47d600b0-d227-441e-997a-7a03837cdb67 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90602025-04-30T06:26:39.761ZINFOtest_disk_expunge (dropshot): request completed latency_us = 521 local_addr = [::1]:44559 method = GET remote_addr = [::1]:53935 req_id = 47d600b0-d227-441e-997a-7a03837cdb67 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
90612025-04-30T06:26:39.762ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
90622025-04-30T06:26:39.762ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "47d600b0-d227-441e-997a-7a03837cdb67", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
90632025-04-30T06:26:39.764ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90642025-04-30T06:26:39.765ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
90652025-04-30T06:26:39.765ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
90662025-04-30T06:26:39.766ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90672025-04-30T06:26:39.766ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90682025-04-30T06:26:39.766ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90692025-04-30T06:26:39.766ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90702025-04-30T06:26:39.767ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90712025-04-30T06:26:39.771ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90722025-04-30T06:26:39.777ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90732025-04-30T06:26:39.777ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90742025-04-30T06:26:39.778ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90752025-04-30T06:26:39.778ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90762025-04-30T06:26:39.816ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90772025-04-30T06:26:39.816ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90782025-04-30T06:26:39.816ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90792025-04-30T06:26:39.816ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
90802025-04-30T06:26:39.817ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90812025-04-30T06:26:39.819ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
90822025-04-30T06:26:39.819ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
90832025-04-30T06:26:39.819ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 381.063714ms iteration = 30 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
90842025-04-30T06:26:39.819ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 31 reason = Signaled
90852025-04-30T06:26:39.819ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
90862025-04-30T06:26:39.819ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90872025-04-30T06:26:39.819ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90882025-04-30T06:26:39.819ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90892025-04-30T06:26:39.820ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
90902025-04-30T06:26:39.821ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90912025-04-30T06:26:39.829ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90922025-04-30T06:26:39.829ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90932025-04-30T06:26:39.829ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
90942025-04-30T06:26:39.830ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90952025-04-30T06:26:39.833ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90962025-04-30T06:26:39.838ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
90972025-04-30T06:26:39.838ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
90982025-04-30T06:26:39.838ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
90992025-04-30T06:26:39.838ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91002025-04-30T06:26:39.840ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91012025-04-30T06:26:39.840ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
91022025-04-30T06:26:39.841ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91032025-04-30T06:26:39.842ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
91042025-04-30T06:26:39.842ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
91052025-04-30T06:26:39.844ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 2b 42 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 +B..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57413 req_id = e60ff45f-27d1-4c9f-9982-551c80722662
91062025-04-30T06:26:39.844ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11074,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57413 req_id = e60ff45f-27d1-4c9f-9982-551c80722662
91072025-04-30T06:26:39.844ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
91082025-04-30T06:26:39.844ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
91092025-04-30T06:26:39.844ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57413 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e60ff45f-27d1-4c9f-9982-551c80722662
91102025-04-30T06:26:39.844ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3264, tv_nsec: 643322087 } })
91112025-04-30T06:26:39.844ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
91122025-04-30T06:26:39.844ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fa 03 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49637 req_id = 402df1b7-688a-42c8-9084-09c5bf9ebdc4
91132025-04-30T06:26:39.844ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64003,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49637 req_id = 402df1b7-688a-42c8-9084-09c5bf9ebdc4
91142025-04-30T06:26:39.844ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
91152025-04-30T06:26:39.845ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49637 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 402df1b7-688a-42c8-9084-09c5bf9ebdc4
91162025-04-30T06:26:39.898ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
91172025-04-30T06:26:39.899ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d7 87 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54429 req_id = 2798e6af-2a24-43c9-a202-565727a86b6f
91182025-04-30T06:26:39.899ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55175,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54429 req_id = 2798e6af-2a24-43c9-a202-565727a86b6f
91192025-04-30T06:26:39.899ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
91202025-04-30T06:26:39.900ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55175, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54429 req_id = 2798e6af-2a24-43c9-a202-565727a86b6f
91212025-04-30T06:26:39.900ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
91222025-04-30T06:26:39.900ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
91232025-04-30T06:26:39.900ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
91242025-04-30T06:26:39.903ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a03bad8-771b-4857-a0a8-1ef405da58b3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91252025-04-30T06:26:39.906ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
91262025-04-30T06:26:39.908ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a64c326-ad4f-4f6c-bc84-eb184bfebd83", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91272025-04-30T06:26:39.908ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
91282025-04-30T06:26:39.913ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0b134ac-78e8-4b03-bbd5-35892f8f9fab", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91292025-04-30T06:26:39.913ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
91302025-04-30T06:26:39.916ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1fdc415-3de5-4d4a-b67c-c5416fb0a5cc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91312025-04-30T06:26:39.917ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
91322025-04-30T06:26:39.920ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f89e32d8-d3e3-4081-8563-2ae718e4e979", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91332025-04-30T06:26:39.920ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
91342025-04-30T06:26:39.923ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46206a0e-0990-45d7-8cb0-30eace99d4e3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91352025-04-30T06:26:39.923ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
91362025-04-30T06:26:39.932ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
91372025-04-30T06:26:39.932ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "08a59db9-4d42-4025-a13d-65fa63f6788e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91382025-04-30T06:26:39.933ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
91392025-04-30T06:26:39.938ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c6f6fec-7411-40f9-ae69-5c5c5c5f4e3c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91402025-04-30T06:26:39.938ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
91412025-04-30T06:26:39.941ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b7ffe5f-14a1-43f1-9b92-8fba2c023a4b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91422025-04-30T06:26:39.941ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
91432025-04-30T06:26:39.942ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e25fd7c-a456-4835-94f6-5ae4d9ecbe94", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91442025-04-30T06:26:39.942ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
91452025-04-30T06:26:39.945ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "473a64bd-9409-4d2b-b83b-f6d32c409870", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91462025-04-30T06:26:39.945ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
91472025-04-30T06:26:39.947ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37f3e76e-8245-4485-a3cc-843d1e0c025b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91482025-04-30T06:26:39.948ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
91492025-04-30T06:26:39.950ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe5d31ae-9fba-4246-9519-52e4411ffc06", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91502025-04-30T06:26:39.951ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
91512025-04-30T06:26:39.953ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59410a17-29bd-48e1-8285-03926de16b79", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91522025-04-30T06:26:39.953ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
91532025-04-30T06:26:39.956ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
91542025-04-30T06:26:39.956ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
91552025-04-30T06:26:39.956ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
91562025-04-30T06:26:39.958ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
91572025-04-30T06:26:39.959ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "22e5de13-032b-4946-b683-6afa7aef0ca0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91582025-04-30T06:26:39.960ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
91592025-04-30T06:26:39.963ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
91602025-04-30T06:26:39.964ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
91612025-04-30T06:26:39.964ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27bf2f1e-18f7-4d96-b6a1-22335d6be295", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91622025-04-30T06:26:39.964ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
91632025-04-30T06:26:39.966ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
91642025-04-30T06:26:39.966ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
91652025-04-30T06:26:39.968ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dd46b8f-4342-4dbc-b879-070ffaaf2964", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91662025-04-30T06:26:39.968ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
91672025-04-30T06:26:39.970ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8afe91a-b3fa-4f5c-899d-664d6b8aab94", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91682025-04-30T06:26:39.970ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
91692025-04-30T06:26:39.973ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77150243-b980-438f-8205-d952f7e66af9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91702025-04-30T06:26:39.973ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
91712025-04-30T06:26:39.974ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cdb0f58-0186-4650-bb70-da1060c1c374", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91722025-04-30T06:26:39.975ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
91732025-04-30T06:26:39.976ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb634096-2edf-4623-8b2f-ba988c1c9787", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91742025-04-30T06:26:39.977ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
91752025-04-30T06:26:39.978ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3830cba-0a10-4adb-816f-332b71edcd86", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91762025-04-30T06:26:39.978ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
91772025-04-30T06:26:39.986ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "325a7d72-a3c2-4590-9cc2-87e89d019e48", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91782025-04-30T06:26:39.986ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
91792025-04-30T06:26:39.986ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91802025-04-30T06:26:39.986ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91812025-04-30T06:26:39.987ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91822025-04-30T06:26:39.987ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
91832025-04-30T06:26:39.987ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91842025-04-30T06:26:39.989ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c08e4859-b957-482c-8e80-239293fc2b89", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91852025-04-30T06:26:39.990ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
91862025-04-30T06:26:39.990ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91872025-04-30T06:26:39.997ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91882025-04-30T06:26:39.997ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91892025-04-30T06:26:39.997ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
91902025-04-30T06:26:39.998ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91912025-04-30T06:26:39.999ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01ac59a9-cb37-4c42-bce1-f57f13adecba", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91922025-04-30T06:26:40.000ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
91932025-04-30T06:26:40.002ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91942025-04-30T06:26:40.003ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
91952025-04-30T06:26:40.003ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91962025-04-30T06:26:40.004ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b29b8b6a-a6fc-4e34-8f96-abeda59f6935", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91972025-04-30T06:26:40.004ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
91982025-04-30T06:26:40.006ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fca1b8f9-221d-4064-988f-0b5e1d472175", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
91992025-04-30T06:26:40.006ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
92002025-04-30T06:26:40.015ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aff62cb7-eb67-4b1f-97a4-23d16cf5bf9f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92012025-04-30T06:26:40.015ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
92022025-04-30T06:26:40.048ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92032025-04-30T06:26:40.055ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92042025-04-30T06:26:40.061ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
92052025-04-30T06:26:40.065ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
92062025-04-30T06:26:40.081ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac68e442-8900-43aa-a13b-7e5adf52be5b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92072025-04-30T06:26:40.083ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
92082025-04-30T06:26:40.084ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92092025-04-30T06:26:40.087ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92102025-04-30T06:26:40.088ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92112025-04-30T06:26:40.088ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
92122025-04-30T06:26:40.106ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5aedd45c-daf8-4204-b60d-7a4c3e98fe67", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92132025-04-30T06:26:40.118ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
92142025-04-30T06:26:40.118ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca393a35-c808-4b13-b4cb-53734071002f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92152025-04-30T06:26:40.118ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
92162025-04-30T06:26:40.118ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92172025-04-30T06:26:40.118ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92182025-04-30T06:26:40.118ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92192025-04-30T06:26:40.118ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
92202025-04-30T06:26:40.118ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92212025-04-30T06:26:40.118ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4dcefc6-9704-488d-b59d-9267df31ba30", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92222025-04-30T06:26:40.118ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
92232025-04-30T06:26:40.121ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92242025-04-30T06:26:40.129ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92252025-04-30T06:26:40.129ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92262025-04-30T06:26:40.129ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
92272025-04-30T06:26:40.130ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92282025-04-30T06:26:40.131ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a49b5e4b-8661-43ac-9e00-902b836e7b7b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92292025-04-30T06:26:40.131ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
92302025-04-30T06:26:40.132ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd7ed9a0-8c7e-49ae-9611-819e2d0f31e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92312025-04-30T06:26:40.133ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
92322025-04-30T06:26:40.133ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92332025-04-30T06:26:40.133ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
92342025-04-30T06:26:40.133ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92352025-04-30T06:26:40.135ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdbe9e6a-08c6-45da-a987-c868e57f91ff", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92362025-04-30T06:26:40.135ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
92372025-04-30T06:26:40.138ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef012f28-ced1-4b81-a2ad-5c05f66bea2f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92382025-04-30T06:26:40.138ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
92392025-04-30T06:26:40.139ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0489784b-34f3-4a81-9027-18297973e1a6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92402025-04-30T06:26:40.139ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
92412025-04-30T06:26:40.141ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5c56a39-8d37-4587-9491-32ed514a4505", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92422025-04-30T06:26:40.141ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
92432025-04-30T06:26:40.143ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6011645a-4348-49ec-bbee-f5892c5a7e4a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92442025-04-30T06:26:40.143ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
92452025-04-30T06:26:40.145ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcd49c80-7dd5-4f7e-ae7f-e0dd4486d74b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92462025-04-30T06:26:40.145ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
92472025-04-30T06:26:40.148ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffccdbb2-fe3e-4cce-add0-3b458498cb9d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92482025-04-30T06:26:40.148ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
92492025-04-30T06:26:40.150ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e7ffd08-caf1-4480-b3a3-e446f06feff7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92502025-04-30T06:26:40.150ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
92512025-04-30T06:26:40.151ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "429fe342-96b7-4af1-b9c1-0296d178dfe2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92522025-04-30T06:26:40.152ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
92532025-04-30T06:26:40.152ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "447060da-d373-4050-b67b-78f4f283ec06", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92542025-04-30T06:26:40.153ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
92552025-04-30T06:26:40.154ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61052104-c00e-4bc5-8dd0-589e45a05dd7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92562025-04-30T06:26:40.154ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92572025-04-30T06:26:40.154ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92582025-04-30T06:26:40.154ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92592025-04-30T06:26:40.154ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92602025-04-30T06:26:40.154ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92612025-04-30T06:26:40.165ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92622025-04-30T06:26:40.171ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92632025-04-30T06:26:40.172ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92642025-04-30T06:26:40.172ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92652025-04-30T06:26:40.172ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92662025-04-30T06:26:40.172ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92672025-04-30T06:26:40.173ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92682025-04-30T06:26:40.174ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
92692025-04-30T06:26:40.175ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92702025-04-30T06:26:40.181ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92712025-04-30T06:26:40.181ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92722025-04-30T06:26:40.181ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92732025-04-30T06:26:40.182ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92742025-04-30T06:26:40.241ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
92752025-04-30T06:26:40.241ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
92762025-04-30T06:26:40.242ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92772025-04-30T06:26:40.242ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92782025-04-30T06:26:40.242ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92792025-04-30T06:26:40.242ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
92802025-04-30T06:26:40.243ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92812025-04-30T06:26:40.243ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:55742 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
92822025-04-30T06:26:40.244ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:55742 req_id = 6fe6a3df-38b3-44ef-9941-0ebac6b28dcf sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92832025-04-30T06:26:40.244ZINFOtest_disk_expunge (dropshot): request completed latency_us = 523 local_addr = [::1]:44559 method = GET remote_addr = [::1]:55742 req_id = 6fe6a3df-38b3-44ef-9941-0ebac6b28dcf response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92842025-04-30T06:26:40.244ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fe6a3df-38b3-44ef-9941-0ebac6b28dcf", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:39 GMT"} })
92852025-04-30T06:26:40.246ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92862025-04-30T06:26:40.246ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92872025-04-30T06:26:40.246ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
92882025-04-30T06:26:40.248ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92892025-04-30T06:26:40.248ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92902025-04-30T06:26:40.248ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92912025-04-30T06:26:40.248ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92922025-04-30T06:26:40.249ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92932025-04-30T06:26:40.249ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92942025-04-30T06:26:40.255ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92952025-04-30T06:26:40.256ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92962025-04-30T06:26:40.256ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
92972025-04-30T06:26:40.256ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92982025-04-30T06:26:40.257ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92992025-04-30T06:26:40.261ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
93002025-04-30T06:26:40.261ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93012025-04-30T06:26:40.261ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93022025-04-30T06:26:40.262ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93032025-04-30T06:26:40.262ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93042025-04-30T06:26:40.262ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
93052025-04-30T06:26:40.263ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93062025-04-30T06:26:40.303ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
93072025-04-30T06:26:40.303ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
93082025-04-30T06:26:40.304ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
93092025-04-30T06:26:40.304ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 484.294195ms iteration = 31 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
93102025-04-30T06:26:40.304ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 32 reason = Signaled
93112025-04-30T06:26:40.304ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
93122025-04-30T06:26:40.304ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93132025-04-30T06:26:40.304ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93142025-04-30T06:26:40.304ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93152025-04-30T06:26:40.305ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93162025-04-30T06:26:40.307ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93172025-04-30T06:26:40.311ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
93182025-04-30T06:26:40.311ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93192025-04-30T06:26:40.311ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93202025-04-30T06:26:40.312ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93212025-04-30T06:26:40.315ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
93222025-04-30T06:26:40.315ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
93232025-04-30T06:26:40.315ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: dc f5 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62601 req_id = b6936e4e-103a-40f1-8e9c-e462f53060f6
93242025-04-30T06:26:40.315ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56565,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62601 req_id = b6936e4e-103a-40f1-8e9c-e462f53060f6
93252025-04-30T06:26:40.316ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
93262025-04-30T06:26:40.316ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93272025-04-30T06:26:40.316ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62601 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b6936e4e-103a-40f1-8e9c-e462f53060f6
93282025-04-30T06:26:40.316ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3265, tv_nsec: 115477606 } })
93292025-04-30T06:26:40.316ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
93302025-04-30T06:26:40.316ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9d b1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58615 req_id = a8357874-430e-485b-89d3-922a47395cf8
93312025-04-30T06:26:40.316ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40369,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58615 req_id = a8357874-430e-485b-89d3-922a47395cf8
93322025-04-30T06:26:40.317ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93332025-04-30T06:26:40.317ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58615 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a8357874-430e-485b-89d3-922a47395cf8
93342025-04-30T06:26:40.368ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
93352025-04-30T06:26:40.368ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93362025-04-30T06:26:40.369ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93372025-04-30T06:26:40.370ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93382025-04-30T06:26:40.370ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
93392025-04-30T06:26:40.371ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93402025-04-30T06:26:40.372ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b0 79 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .y..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55577 req_id = 901fae77-9fda-48ef-8a70-07d7af8a3921
93412025-04-30T06:26:40.372ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45177,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55577 req_id = 901fae77-9fda-48ef-8a70-07d7af8a3921
93422025-04-30T06:26:40.372ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
93432025-04-30T06:26:40.372ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45177, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55577 req_id = 901fae77-9fda-48ef-8a70-07d7af8a3921
93442025-04-30T06:26:40.373ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
93452025-04-30T06:26:40.373ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
93462025-04-30T06:26:40.373ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
93472025-04-30T06:26:40.374ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93482025-04-30T06:26:40.384ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
93492025-04-30T06:26:40.384ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93502025-04-30T06:26:40.384ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
93512025-04-30T06:26:40.384ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93522025-04-30T06:26:40.387ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5088c3c4-ec52-4f87-b62c-9de0e6bcd203", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93532025-04-30T06:26:40.387ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
93542025-04-30T06:26:40.388ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93552025-04-30T06:26:40.388ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
93562025-04-30T06:26:40.388ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93572025-04-30T06:26:40.390ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80c34246-83e5-469e-8e2d-0e39885d92f7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93582025-04-30T06:26:40.390ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
93592025-04-30T06:26:40.394ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c0b78e1-874a-4bd6-b558-f548435e3ca7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93602025-04-30T06:26:40.394ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
93612025-04-30T06:26:40.398ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52e26c43-35e6-439b-bb0d-770e8dc19a7a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93622025-04-30T06:26:40.398ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
93632025-04-30T06:26:40.401ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7d594d2-34ea-4200-bf46-aa00724b1752", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93642025-04-30T06:26:40.401ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
93652025-04-30T06:26:40.403ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5a66f71-a7a0-4ca7-ab86-07d183ccddaf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93662025-04-30T06:26:40.404ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
93672025-04-30T06:26:40.407ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85efc724-a908-4aef-9a3d-c511daff4df9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93682025-04-30T06:26:40.407ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
93692025-04-30T06:26:40.409ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea970179-2884-4751-a85b-bfb9e8e00206", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93702025-04-30T06:26:40.409ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
93712025-04-30T06:26:40.411ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31d35869-4360-4687-8bc1-882dd339e4b1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93722025-04-30T06:26:40.411ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
93732025-04-30T06:26:40.412ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f5f6538-e9b0-4392-a9a3-501c7760ad74", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93742025-04-30T06:26:40.412ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
93752025-04-30T06:26:40.414ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bfc48ad-eacf-48bd-ae2b-180f66d40b2f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93762025-04-30T06:26:40.414ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
93772025-04-30T06:26:40.415ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
93782025-04-30T06:26:40.416ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ea61a20-98e1-4019-8904-a318625272dc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93792025-04-30T06:26:40.416ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
93802025-04-30T06:26:40.417ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aec95f9f-ebe3-4fb9-81ea-388fd8e66f40", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93812025-04-30T06:26:40.418ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
93822025-04-30T06:26:40.419ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f965708c-3336-4bec-b554-33e24df17326", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93832025-04-30T06:26:40.420ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
93842025-04-30T06:26:40.421ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5be7a0f-aa15-454f-86ca-b1d7064f32d0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93852025-04-30T06:26:40.421ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
93862025-04-30T06:26:40.424ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa5e987f-9b1b-4b60-8875-631cab9ab598", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93872025-04-30T06:26:40.424ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
93882025-04-30T06:26:40.425ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35bec248-4f9a-4a77-8ce3-8812e67cfe82", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93892025-04-30T06:26:40.426ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
93902025-04-30T06:26:40.427ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98cdd16d-1b11-43c1-8ac2-da188b8bb880", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93912025-04-30T06:26:40.428ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
93922025-04-30T06:26:40.437ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b43d4e46-6c72-4a7b-9413-059e594b7c48", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93932025-04-30T06:26:40.437ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
93942025-04-30T06:26:40.441ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89e48ac2-8f0c-4f6c-b70c-7a2bbb7fca3b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93952025-04-30T06:26:40.441ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
93962025-04-30T06:26:40.442ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b301f5e5-b4fb-4092-bd4d-27f3e9bdb370", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93972025-04-30T06:26:40.443ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
93982025-04-30T06:26:40.444ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db3cf4bb-cb6f-4c04-b565-cf60327f615b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
93992025-04-30T06:26:40.444ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
94002025-04-30T06:26:40.446ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0d28187-98c2-4a73-bfbf-f675c83834e1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94012025-04-30T06:26:40.446ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
94022025-04-30T06:26:40.449ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c27b8888-b376-4302-b8f7-df2f8e9ed8a6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94032025-04-30T06:26:40.449ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
94042025-04-30T06:26:40.452ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "defa8259-1a04-4ef6-be5f-3221a55c5b03", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94052025-04-30T06:26:40.452ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
94062025-04-30T06:26:40.461ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bef2718-6e4d-4d72-9f9f-6fc5f8efe23b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94072025-04-30T06:26:40.462ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
94082025-04-30T06:26:40.506ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94092025-04-30T06:26:40.506ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94102025-04-30T06:26:40.506ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94112025-04-30T06:26:40.506ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
94122025-04-30T06:26:40.511ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94132025-04-30T06:26:40.515ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94142025-04-30T06:26:40.523ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94152025-04-30T06:26:40.523ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94162025-04-30T06:26:40.523ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
94172025-04-30T06:26:40.523ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94182025-04-30T06:26:40.525ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea08d9f5-c251-4587-934d-20c106927e87", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94192025-04-30T06:26:40.526ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
94202025-04-30T06:26:40.526ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94212025-04-30T06:26:40.527ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
94222025-04-30T06:26:40.527ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94232025-04-30T06:26:40.532ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ede9f2cf-2966-4f16-9c99-eb61538a34b4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94242025-04-30T06:26:40.532ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
94252025-04-30T06:26:40.537ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13c16eb3-2791-4ac9-b95d-69d7963abc3d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94262025-04-30T06:26:40.537ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
94272025-04-30T06:26:40.544ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05431eeb-05d6-4134-88b3-bc8a9b46b2fc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94282025-04-30T06:26:40.544ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
94292025-04-30T06:26:40.548ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb40fc51-d420-442e-906d-8bff6512f0a1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94302025-04-30T06:26:40.549ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
94312025-04-30T06:26:40.550ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a165a084-b23f-4988-b748-d950d6527d70", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94322025-04-30T06:26:40.550ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
94332025-04-30T06:26:40.554ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d53ed6c7-ee6b-4b7d-9b8a-2a6e2113f900", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94342025-04-30T06:26:40.554ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
94352025-04-30T06:26:40.586ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "694ea670-194a-44ad-84e8-ceb08c3c3c94", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94362025-04-30T06:26:40.587ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
94372025-04-30T06:26:40.624ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7aa0ba8-afac-4fcd-985f-eba37d0d6bf6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94382025-04-30T06:26:40.624ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
94392025-04-30T06:26:40.625ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
94402025-04-30T06:26:40.630ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b20f9d7-f675-4368-93f6-cb71294aa535", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94412025-04-30T06:26:40.630ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
94422025-04-30T06:26:40.646ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3f2373f-6cea-4993-a3ea-36d07dc64f05", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94432025-04-30T06:26:40.646ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
94442025-04-30T06:26:40.651ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c78199d-bd63-46fd-962e-f10cfe57b9e8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94452025-04-30T06:26:40.651ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
94462025-04-30T06:26:40.656ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "178eb01a-312d-4a3e-a476-68404b411717", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94472025-04-30T06:26:40.656ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
94482025-04-30T06:26:40.661ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97ff66e4-6bb9-41f8-abf5-a9954449df9b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94492025-04-30T06:26:40.661ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
94502025-04-30T06:26:40.663ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "377d42aa-2905-4686-a47a-b51744a5786c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94512025-04-30T06:26:40.663ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
94522025-04-30T06:26:40.665ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ad86eee-085d-4b70-b135-c74fc4950cc2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94532025-04-30T06:26:40.665ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
94542025-04-30T06:26:40.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c0fa95c-0c0b-412f-b9f0-d43f2254deca", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94552025-04-30T06:26:40.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
94562025-04-30T06:26:40.669ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "40890df1-1f64-4ebd-8283-6302a56e6528", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94572025-04-30T06:26:40.669ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
94582025-04-30T06:26:40.670ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee10ac73-9c1d-4f6c-9509-0a3c187c1fe3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
94592025-04-30T06:26:40.670ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94602025-04-30T06:26:40.670ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94612025-04-30T06:26:40.670ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94622025-04-30T06:26:40.670ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94632025-04-30T06:26:40.671ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94642025-04-30T06:26:40.675ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94652025-04-30T06:26:40.684ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94662025-04-30T06:26:40.684ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94672025-04-30T06:26:40.684ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94682025-04-30T06:26:40.684ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94692025-04-30T06:26:40.684ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94702025-04-30T06:26:40.685ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94712025-04-30T06:26:40.685ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94722025-04-30T06:26:40.685ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94732025-04-30T06:26:40.685ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94742025-04-30T06:26:40.685ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
94752025-04-30T06:26:40.686ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94762025-04-30T06:26:40.689ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94772025-04-30T06:26:40.716ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94782025-04-30T06:26:40.716ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94792025-04-30T06:26:40.716ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
94802025-04-30T06:26:40.726ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94812025-04-30T06:26:40.727ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94822025-04-30T06:26:40.735ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94832025-04-30T06:26:40.735ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94842025-04-30T06:26:40.735ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94852025-04-30T06:26:40.736ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94862025-04-30T06:26:40.802ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94872025-04-30T06:26:40.802ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
94882025-04-30T06:26:40.803ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94892025-04-30T06:26:40.912ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
94902025-04-30T06:26:40.912ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
94912025-04-30T06:26:40.913ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:33850 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
94922025-04-30T06:26:40.913ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
94932025-04-30T06:26:40.915ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
94942025-04-30T06:26:40.915ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 3dfd2f8f-516a-4764-9115-1fc1dde84ba9 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
94952025-04-30T06:26:40.916ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:33850 req_id = ef6041c8-28dc-4fb1-95ee-e69f8ab26c4d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
94962025-04-30T06:26:40.916ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 3dfd2f8f-516a-4764-9115-1fc1dde84ba9 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
94972025-04-30T06:26:40.916ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 3dfd2f8f-516a-4764-9115-1fc1dde84ba9 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
94982025-04-30T06:26:40.918ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 3dfd2f8f-516a-4764-9115-1fc1dde84ba9 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
94992025-04-30T06:26:40.918ZINFOtest_disk_expunge (dropshot): request completed latency_us = 2662 local_addr = [::1]:44559 method = GET remote_addr = [::1]:33850 req_id = ef6041c8-28dc-4fb1-95ee-e69f8ab26c4d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
95002025-04-30T06:26:40.919ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef6041c8-28dc-4fb1-95ee-e69f8ab26c4d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
95012025-04-30T06:26:40.923ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
95022025-04-30T06:26:40.923ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
95032025-04-30T06:26:40.923ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
95042025-04-30T06:26:40.927ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
95052025-04-30T06:26:40.929ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95062025-04-30T06:26:40.930ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95072025-04-30T06:26:40.930ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95082025-04-30T06:26:40.931ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95092025-04-30T06:26:40.937ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 21773 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 3dfd2f8f-516a-4764-9115-1fc1dde84ba9 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
95102025-04-30T06:26:40.940ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95112025-04-30T06:26:40.951ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
95122025-04-30T06:26:40.952ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95132025-04-30T06:26:40.952ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95142025-04-30T06:26:40.953ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95152025-04-30T06:26:40.954ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "3dfd2f8f-516a-4764-9115-1fc1dde84ba9", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
95162025-04-30T06:26:40.954ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95172025-04-30T06:26:40.956ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = eb30a6ab-0e3d-4be0-b22d-13d7c8cdf2a5 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95182025-04-30T06:26:40.958ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = eb30a6ab-0e3d-4be0-b22d-13d7c8cdf2a5 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95192025-04-30T06:26:40.959ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = eb30a6ab-0e3d-4be0-b22d-13d7c8cdf2a5 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95202025-04-30T06:26:40.960ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = eb30a6ab-0e3d-4be0-b22d-13d7c8cdf2a5 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95212025-04-30T06:26:40.962ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
95222025-04-30T06:26:40.962ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
95232025-04-30T06:26:40.963ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
95242025-04-30T06:26:40.964ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
95252025-04-30T06:26:40.964ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 8163 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = eb30a6ab-0e3d-4be0-b22d-13d7c8cdf2a5 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
95262025-04-30T06:26:40.973ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb30a6ab-0e3d-4be0-b22d-13d7c8cdf2a5", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
95272025-04-30T06:26:40.973ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
95282025-04-30T06:26:40.974ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95292025-04-30T06:26:40.974ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
95302025-04-30T06:26:40.975ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
95312025-04-30T06:26:40.975ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
95322025-04-30T06:26:40.982ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
95332025-04-30T06:26:40.982ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
95342025-04-30T06:26:41.014ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
95352025-04-30T06:26:41.047ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95362025-04-30T06:26:41.047ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95372025-04-30T06:26:41.054ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95382025-04-30T06:26:41.055ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95392025-04-30T06:26:41.056ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95402025-04-30T06:26:41.056ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95412025-04-30T06:26:41.065ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95422025-04-30T06:26:41.067ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95432025-04-30T06:26:41.067ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95442025-04-30T06:26:41.067ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
95452025-04-30T06:26:41.069ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95462025-04-30T06:26:41.070ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
95472025-04-30T06:26:41.075ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95482025-04-30T06:26:41.086ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95492025-04-30T06:26:41.086ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95502025-04-30T06:26:41.086ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
95512025-04-30T06:26:41.087ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95522025-04-30T06:26:41.102ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
95532025-04-30T06:26:41.103ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
95542025-04-30T06:26:41.104ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 800.571166ms iteration = 32 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
95552025-04-30T06:26:41.105ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 33 reason = Signaled
95562025-04-30T06:26:41.105ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
95572025-04-30T06:26:41.105ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95582025-04-30T06:26:41.105ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95592025-04-30T06:26:41.105ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95602025-04-30T06:26:41.106ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95612025-04-30T06:26:41.109ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95622025-04-30T06:26:41.109ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
95632025-04-30T06:26:41.113ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95642025-04-30T06:26:41.116ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95652025-04-30T06:26:41.123ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
95662025-04-30T06:26:41.123ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95672025-04-30T06:26:41.124ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95682025-04-30T06:26:41.124ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95692025-04-30T06:26:41.132ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
95702025-04-30T06:26:41.132ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
95712025-04-30T06:26:41.137ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6a 42 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 jB..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57424 req_id = 3800dd92-bc1d-4e56-bed5-f9bc55922ce5
95722025-04-30T06:26:41.137ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27202,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57424 req_id = 3800dd92-bc1d-4e56-bed5-f9bc55922ce5
95732025-04-30T06:26:41.139ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
95742025-04-30T06:26:41.142ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95752025-04-30T06:26:41.142ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57424 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3800dd92-bc1d-4e56-bed5-f9bc55922ce5
95762025-04-30T06:26:41.142ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3265, tv_nsec: 941067330 } })
95772025-04-30T06:26:41.142ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
95782025-04-30T06:26:41.142ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7f c3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59168 req_id = 34905d2a-0a26-4565-8859-9304b6037afb
95792025-04-30T06:26:41.143ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32707,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59168 req_id = 34905d2a-0a26-4565-8859-9304b6037afb
95802025-04-30T06:26:41.145ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95812025-04-30T06:26:41.145ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59168 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 34905d2a-0a26-4565-8859-9304b6037afb
95822025-04-30T06:26:41.200ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
95832025-04-30T06:26:41.202ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 77 21 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c w!..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60831 req_id = 7c71eeb1-9617-4613-bec0-73ebe282c9c2
95842025-04-30T06:26:41.204ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30497,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60831 req_id = 7c71eeb1-9617-4613-bec0-73ebe282c9c2
95852025-04-30T06:26:41.205ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
95862025-04-30T06:26:41.205ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30497, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60831 req_id = 7c71eeb1-9617-4613-bec0-73ebe282c9c2
95872025-04-30T06:26:41.206ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
95882025-04-30T06:26:41.206ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
95892025-04-30T06:26:41.206ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
95902025-04-30T06:26:41.210ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e7e6169-b542-48c7-aff9-d8eab76e3995", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
95912025-04-30T06:26:41.211ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
95922025-04-30T06:26:41.215ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41a648a8-2598-4ad0-a007-15ec1101b26c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
95932025-04-30T06:26:41.215ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
95942025-04-30T06:26:41.222ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98b2ed3f-5ea4-45f8-9322-49fb97a65ef3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
95952025-04-30T06:26:41.222ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
95962025-04-30T06:26:41.232ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
95972025-04-30T06:26:41.235ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3bc934f-da8e-408e-b01e-eeb65b646851", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
95982025-04-30T06:26:41.236ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
95992025-04-30T06:26:41.239ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83cf4407-ca5d-4cfd-98bb-726bfef35472", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96002025-04-30T06:26:41.239ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
96012025-04-30T06:26:41.246ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c74a7292-a08e-439e-8ae2-f61972bdd020", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96022025-04-30T06:26:41.246ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
96032025-04-30T06:26:41.251ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63f4fd43-0da7-46bc-ba99-5ba25a8789c3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96042025-04-30T06:26:41.251ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
96052025-04-30T06:26:41.260ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "438d097b-0539-4a24-bb24-ef2e0184f09d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96062025-04-30T06:26:41.260ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
96072025-04-30T06:26:41.262ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2eacbed1-5b94-40a2-a3b0-394b2b76f19c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96082025-04-30T06:26:41.262ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
96092025-04-30T06:26:41.263ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bac3b23f-2588-443d-bd74-b8a4df239b88", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96102025-04-30T06:26:41.263ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
96112025-04-30T06:26:41.264ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8620cef-1d48-4e24-bc9a-abed51df07ae", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96122025-04-30T06:26:41.265ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
96132025-04-30T06:26:41.266ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d10f787-daa1-41bd-8ff5-fb88b86f3eb1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96142025-04-30T06:26:41.266ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
96152025-04-30T06:26:41.269ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8af6316b-d0ed-41d0-800a-0833c5eb2484", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96162025-04-30T06:26:41.270ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
96172025-04-30T06:26:41.282ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96182025-04-30T06:26:41.283ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96192025-04-30T06:26:41.283ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96202025-04-30T06:26:41.283ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
96212025-04-30T06:26:41.285ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96222025-04-30T06:26:41.286ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99f73451-a28b-4dc8-a07c-1a7936766da2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96232025-04-30T06:26:41.286ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
96242025-04-30T06:26:41.288ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96252025-04-30T06:26:41.297ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96262025-04-30T06:26:41.297ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96272025-04-30T06:26:41.297ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
96282025-04-30T06:26:41.297ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96292025-04-30T06:26:41.299ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9786328-7346-4503-98ae-f9414a2463e9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96302025-04-30T06:26:41.299ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
96312025-04-30T06:26:41.300ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96322025-04-30T06:26:41.300ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
96332025-04-30T06:26:41.301ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96342025-04-30T06:26:41.303ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3753f744-c1b3-49bc-9b45-75f64f6720e9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96352025-04-30T06:26:41.304ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
96362025-04-30T06:26:41.307ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b34583e-b1ff-4c19-89df-93ae39b50416", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96372025-04-30T06:26:41.307ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
96382025-04-30T06:26:41.309ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f522d097-6b33-4b56-a50b-d45660ca37de", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96392025-04-30T06:26:41.309ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
96402025-04-30T06:26:41.315ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8916f22-0d68-45b3-ad26-456b0d23b6a8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96412025-04-30T06:26:41.315ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
96422025-04-30T06:26:41.316ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eaa89858-d69e-4dd1-843d-f13304fda5f7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96432025-04-30T06:26:41.316ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
96442025-04-30T06:26:41.317ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "417d2e1a-64d1-449d-96d1-668f607dae61", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96452025-04-30T06:26:41.318ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
96462025-04-30T06:26:41.319ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79e4ecdd-72f8-4bd5-9e12-1d57f9c4dd23", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96472025-04-30T06:26:41.319ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
96482025-04-30T06:26:41.321ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5132d1a-5242-431a-b80f-ae780396f32f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96492025-04-30T06:26:41.321ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
96502025-04-30T06:26:41.328ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6627980-194c-4789-bf1d-8dc863ebf8c8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96512025-04-30T06:26:41.329ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
96522025-04-30T06:26:41.333ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4af71d3c-6981-46d5-a797-1768dc6d9fa8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96532025-04-30T06:26:41.333ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
96542025-04-30T06:26:41.336ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87ea41b1-ec66-44a1-be08-27aaec7760b3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96552025-04-30T06:26:41.336ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
96562025-04-30T06:26:41.341ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
96572025-04-30T06:26:41.344ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c896b8d5-4dc2-492a-b08b-ee9210e1f26f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96582025-04-30T06:26:41.344ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
96592025-04-30T06:26:41.348ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ac42ad7-78c5-4b33-9e25-d875b0006186", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96602025-04-30T06:26:41.349ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
96612025-04-30T06:26:41.351ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f21feb2f-2064-420d-a9c0-e875bafc12df", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96622025-04-30T06:26:41.352ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
96632025-04-30T06:26:41.353ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34b4c65d-4a83-4839-8e13-4becd6165619", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96642025-04-30T06:26:41.354ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
96652025-04-30T06:26:41.355ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "226f3f1a-a8d0-4c1c-b2b4-2e12ae540b6f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96662025-04-30T06:26:41.355ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
96672025-04-30T06:26:41.356ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be916869-33f6-4eed-885f-27cad308ee38", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96682025-04-30T06:26:41.356ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
96692025-04-30T06:26:41.358ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9de1c33-4fca-4133-82bd-275169992a24", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96702025-04-30T06:26:41.358ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
96712025-04-30T06:26:41.360ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7365b29-0c7c-4de4-9da7-f5a0e68db077", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96722025-04-30T06:26:41.360ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
96732025-04-30T06:26:41.363ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a4e0c02-3c1e-494f-96f0-2a5849a23dc7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96742025-04-30T06:26:41.364ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
96752025-04-30T06:26:41.369ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d47f908-41a5-4ce5-a607-3c8849ed0d4f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96762025-04-30T06:26:41.369ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
96772025-04-30T06:26:41.372ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f90753c5-52ae-4fc9-b3f9-2346ad2bffdd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96782025-04-30T06:26:41.373ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
96792025-04-30T06:26:41.376ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c998894-2ea0-480a-9f18-ab1a383d5c35", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96802025-04-30T06:26:41.376ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
96812025-04-30T06:26:41.385ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32d33fac-70a3-454c-ad89-d640034074bb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:40 GMT"} })
96822025-04-30T06:26:41.387ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
96832025-04-30T06:26:41.388ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17bbcbfa-4bed-4bc8-81f6-f8c371f5c633", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
96842025-04-30T06:26:41.388ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
96852025-04-30T06:26:41.390ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "44cfeed0-7a63-49e4-9b63-8a6fb84b8b86", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
96862025-04-30T06:26:41.390ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
96872025-04-30T06:26:41.391ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45c50706-5121-4cd9-8a11-fde7cfbda465", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
96882025-04-30T06:26:41.391ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
96892025-04-30T06:26:41.392ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "306f3e04-53ca-495d-a33f-a4f3c1b139a1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
96902025-04-30T06:26:41.392ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
96912025-04-30T06:26:41.392ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96922025-04-30T06:26:41.393ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96932025-04-30T06:26:41.393ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96942025-04-30T06:26:41.393ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
96952025-04-30T06:26:41.393ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96962025-04-30T06:26:41.395ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3aba059-c80b-4aee-94d4-e1521f125a3b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
96972025-04-30T06:26:41.395ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
96982025-04-30T06:26:41.397ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96992025-04-30T06:26:41.405ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97002025-04-30T06:26:41.405ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97012025-04-30T06:26:41.405ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
97022025-04-30T06:26:41.406ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97032025-04-30T06:26:41.406ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1270c536-bb0f-441a-affc-e70d75ca9eb7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
97042025-04-30T06:26:41.407ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
97052025-04-30T06:26:41.407ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97062025-04-30T06:26:41.407ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97072025-04-30T06:26:41.407ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97082025-04-30T06:26:41.407ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97092025-04-30T06:26:41.409ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97102025-04-30T06:26:41.409ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
97112025-04-30T06:26:41.410ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97122025-04-30T06:26:41.411ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97132025-04-30T06:26:41.416ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
97142025-04-30T06:26:41.416ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
97152025-04-30T06:26:41.417ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97162025-04-30T06:26:41.417ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97172025-04-30T06:26:41.417ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97182025-04-30T06:26:41.419ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97192025-04-30T06:26:41.422ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97202025-04-30T06:26:41.428ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
97212025-04-30T06:26:41.428ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97222025-04-30T06:26:41.428ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97232025-04-30T06:26:41.429ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97242025-04-30T06:26:41.518ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
97252025-04-30T06:26:41.518ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
97262025-04-30T06:26:41.519ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:44691 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
97272025-04-30T06:26:41.521ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:44691 req_id = 05e47040-f386-41ee-830d-f55ec2ceb757 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
97282025-04-30T06:26:41.522ZINFOtest_disk_expunge (dropshot): request completed latency_us = 450 local_addr = [::1]:44559 method = GET remote_addr = [::1]:44691 req_id = 05e47040-f386-41ee-830d-f55ec2ceb757 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
97292025-04-30T06:26:41.522ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "05e47040-f386-41ee-830d-f55ec2ceb757", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
97302025-04-30T06:26:41.525ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97312025-04-30T06:26:41.525ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97322025-04-30T06:26:41.525ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
97332025-04-30T06:26:41.528ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
97342025-04-30T06:26:41.528ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97352025-04-30T06:26:41.528ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97362025-04-30T06:26:41.528ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97372025-04-30T06:26:41.528ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97382025-04-30T06:26:41.531ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97392025-04-30T06:26:41.538ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97402025-04-30T06:26:41.539ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97412025-04-30T06:26:41.539ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97422025-04-30T06:26:41.541ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97432025-04-30T06:26:41.557ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
97442025-04-30T06:26:41.588ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
97452025-04-30T06:26:41.588ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
97462025-04-30T06:26:41.589ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 483.743127ms iteration = 33 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
97472025-04-30T06:26:41.589ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 34 reason = Signaled
97482025-04-30T06:26:41.589ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
97492025-04-30T06:26:41.589ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97502025-04-30T06:26:41.589ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97512025-04-30T06:26:41.589ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97522025-04-30T06:26:41.592ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97532025-04-30T06:26:41.597ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97542025-04-30T06:26:41.604ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97552025-04-30T06:26:41.604ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97562025-04-30T06:26:41.604ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97572025-04-30T06:26:41.605ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97582025-04-30T06:26:41.609ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97592025-04-30T06:26:41.609ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97602025-04-30T06:26:41.609ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97612025-04-30T06:26:41.609ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
97622025-04-30T06:26:41.610ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97632025-04-30T06:26:41.610ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
97642025-04-30T06:26:41.611ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
97652025-04-30T06:26:41.612ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 8f f4 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53223 req_id = adc1c324-43cd-4a5e-902a-9a55258fea57
97662025-04-30T06:26:41.612ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36852,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53223 req_id = adc1c324-43cd-4a5e-902a-9a55258fea57
97672025-04-30T06:26:41.612ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
97682025-04-30T06:26:41.612ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97692025-04-30T06:26:41.612ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53223 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = adc1c324-43cd-4a5e-902a-9a55258fea57
97702025-04-30T06:26:41.613ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3266, tv_nsec: 412372965 } })
97712025-04-30T06:26:41.613ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
97722025-04-30T06:26:41.613ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7e 92 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ~...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59956 req_id = f1e9378b-2b11-464b-8e91-95766fc4733a
97732025-04-30T06:26:41.614ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32402,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59956 req_id = f1e9378b-2b11-464b-8e91-95766fc4733a
97742025-04-30T06:26:41.614ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
97752025-04-30T06:26:41.614ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59956 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f1e9378b-2b11-464b-8e91-95766fc4733a
97762025-04-30T06:26:41.705ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
97772025-04-30T06:26:41.706ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97782025-04-30T06:26:41.717ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97792025-04-30T06:26:41.717ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97802025-04-30T06:26:41.717ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
97812025-04-30T06:26:41.717ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97822025-04-30T06:26:41.718ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a0 56 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .V..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61757 req_id = 937f1701-9497-436a-b6fa-78ee9f5ab311
97832025-04-30T06:26:41.718ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41046,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61757 req_id = 937f1701-9497-436a-b6fa-78ee9f5ab311
97842025-04-30T06:26:41.719ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
97852025-04-30T06:26:41.719ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41046, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61757 req_id = 937f1701-9497-436a-b6fa-78ee9f5ab311
97862025-04-30T06:26:41.720ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
97872025-04-30T06:26:41.720ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
97882025-04-30T06:26:41.720ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
97892025-04-30T06:26:41.721ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97902025-04-30T06:26:41.721ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
97912025-04-30T06:26:41.722ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97922025-04-30T06:26:41.729ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd57b90b-634e-4187-8040-0cbaddc0a5f1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
97932025-04-30T06:26:41.732ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
97942025-04-30T06:26:41.734ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02a05522-a865-4256-8531-e709e817f158", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
97952025-04-30T06:26:41.734ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
97962025-04-30T06:26:41.737ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1401d21-c154-4c88-9283-99ed1e8ff17a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
97972025-04-30T06:26:41.737ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
97982025-04-30T06:26:41.739ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8de93fa-aade-441c-9926-ee880516192c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
97992025-04-30T06:26:41.740ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
98002025-04-30T06:26:41.744ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c320441-02a2-427b-840c-e1d529462938", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98012025-04-30T06:26:41.744ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
98022025-04-30T06:26:41.747ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94475837-520c-45bd-9e15-15c7e9eaa142", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98032025-04-30T06:26:41.747ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
98042025-04-30T06:26:41.759ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c5a6302-4eda-4378-8f29-a42811d1c7a8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98052025-04-30T06:26:41.761ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
98062025-04-30T06:26:41.764ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aeedca6c-6e37-46ef-852e-09910232c3b0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98072025-04-30T06:26:41.764ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
98082025-04-30T06:26:41.766ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ef951b0-c99d-4496-b3c6-321c6d527f1f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98092025-04-30T06:26:41.766ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
98102025-04-30T06:26:41.767ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
98112025-04-30T06:26:41.770ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f385ae4-6325-4f74-891c-350c91660fda", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98122025-04-30T06:26:41.770ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
98132025-04-30T06:26:41.775ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e5031a5-934e-4cc3-a15d-2cac9f5278d2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98142025-04-30T06:26:41.775ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
98152025-04-30T06:26:41.777ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da62a8e3-0a14-45ab-82a4-c165501ae94c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98162025-04-30T06:26:41.777ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
98172025-04-30T06:26:41.779ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71d2130f-ff40-4086-aa07-98da422e4f3b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98182025-04-30T06:26:41.779ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
98192025-04-30T06:26:41.786ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e1db046-2a38-417a-95c1-80b067e63ddb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98202025-04-30T06:26:41.786ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
98212025-04-30T06:26:41.789ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9eecae05-0f2f-40cf-8f50-3fe6628c0df6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98222025-04-30T06:26:41.789ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
98232025-04-30T06:26:41.793ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e96b61b-911b-4fbc-804b-d09113243e66", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98242025-04-30T06:26:41.793ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
98252025-04-30T06:26:41.797ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "886e382c-d3b9-4eeb-a7a4-16db7f8efec7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98262025-04-30T06:26:41.797ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
98272025-04-30T06:26:41.801ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6f3966c-874d-4131-b07f-98142cc0216d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98282025-04-30T06:26:41.801ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
98292025-04-30T06:26:41.813ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa267320-e45d-4011-ac31-91c9000d9453", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98302025-04-30T06:26:41.813ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
98312025-04-30T06:26:41.814ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a88f546-7b5f-4d9b-a007-2ba40c4df020", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98322025-04-30T06:26:41.814ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
98332025-04-30T06:26:41.816ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94489995-2bd3-46c6-87a6-8636a6eaadc6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98342025-04-30T06:26:41.816ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
98352025-04-30T06:26:41.817ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98362025-04-30T06:26:41.818ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98372025-04-30T06:26:41.818ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98382025-04-30T06:26:41.818ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
98392025-04-30T06:26:41.819ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98402025-04-30T06:26:41.819ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51f7b32b-f65f-4bae-8bfb-fd17be836745", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98412025-04-30T06:26:41.819ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
98422025-04-30T06:26:41.821ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79637b74-c51b-4d4f-afaa-da4976c4c169", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98432025-04-30T06:26:41.822ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
98442025-04-30T06:26:41.823ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98452025-04-30T06:26:41.829ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98462025-04-30T06:26:41.829ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98472025-04-30T06:26:41.829ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
98482025-04-30T06:26:41.831ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98492025-04-30T06:26:41.834ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9f8a56c-e372-48f6-a1ad-ddc4c89e0700", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98502025-04-30T06:26:41.834ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
98512025-04-30T06:26:41.835ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98522025-04-30T06:26:41.835ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
98532025-04-30T06:26:41.836ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98542025-04-30T06:26:41.841ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebec9c7c-b1cc-44f0-8658-c967d6771d18", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98552025-04-30T06:26:41.841ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
98562025-04-30T06:26:41.844ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60791c56-a48e-41e2-8ae3-7c218acb0d54", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98572025-04-30T06:26:41.845ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
98582025-04-30T06:26:41.849ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "918a2564-0c2a-4392-b674-5d6a519abb53", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98592025-04-30T06:26:41.849ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
98602025-04-30T06:26:41.855ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77919a5a-786d-4604-a59a-6201a25955bf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98612025-04-30T06:26:41.855ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
98622025-04-30T06:26:41.861ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fb72a7e-cb9f-446d-aee4-7141087371a3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98632025-04-30T06:26:41.863ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
98642025-04-30T06:26:41.870ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
98652025-04-30T06:26:41.873ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8d162df-ed8c-4507-9b77-bc8777ad6f2a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98662025-04-30T06:26:41.873ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
98672025-04-30T06:26:41.878ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "439a46ae-85f2-4c18-b756-5615f061bb0a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98682025-04-30T06:26:41.878ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
98692025-04-30T06:26:41.882ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe8fbab6-e59a-4b5a-8f6e-079aa890890f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98702025-04-30T06:26:41.882ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
98712025-04-30T06:26:41.882ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f35d0f8-9d80-4a96-80d2-d95954438f83", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98722025-04-30T06:26:41.882ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
98732025-04-30T06:26:41.882ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "edb747db-6c2a-471b-b7a7-b6c7d9dd31ff", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98742025-04-30T06:26:41.882ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
98752025-04-30T06:26:41.882ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4242150-943f-4085-b5aa-3bf6745c1b48", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98762025-04-30T06:26:41.882ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
98772025-04-30T06:26:41.889ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67a7fa14-dd16-48c9-9517-3e126267cfb3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98782025-04-30T06:26:41.890ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
98792025-04-30T06:26:41.900ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8f9dc8c-1c5a-475f-befa-0e305f26ff9d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98802025-04-30T06:26:41.900ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
98812025-04-30T06:26:41.904ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76d515ff-5243-494a-a57f-90ec312db32f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98822025-04-30T06:26:41.904ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
98832025-04-30T06:26:41.910ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "79470d9b-59b0-4c1c-8fa0-baf61d14e0c4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98842025-04-30T06:26:41.912ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
98852025-04-30T06:26:41.915ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be76c0b1-2cca-4f56-a2d0-ddb48335f32e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98862025-04-30T06:26:41.915ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
98872025-04-30T06:26:41.917ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83514cb0-abbf-4dfc-bda0-d0ad813bec23", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98882025-04-30T06:26:41.917ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
98892025-04-30T06:26:41.920ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fbdec64-f6ca-4cf8-ae6b-42b009f26a9f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98902025-04-30T06:26:41.920ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
98912025-04-30T06:26:41.921ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98922025-04-30T06:26:41.924ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98932025-04-30T06:26:41.924ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98942025-04-30T06:26:41.924ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
98952025-04-30T06:26:41.925ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98962025-04-30T06:26:41.927ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c6e381a-d4fa-4624-95ae-e5390936e00a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
98972025-04-30T06:26:41.927ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
98982025-04-30T06:26:41.929ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98992025-04-30T06:26:41.948ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99002025-04-30T06:26:41.948ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99012025-04-30T06:26:41.948ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
99022025-04-30T06:26:41.950ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99032025-04-30T06:26:41.950ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1cfa5f2-bbcf-4c78-ab27-839b762905a8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
99042025-04-30T06:26:41.950ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
99052025-04-30T06:26:41.951ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99062025-04-30T06:26:41.951ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
99072025-04-30T06:26:41.952ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99082025-04-30T06:26:41.952ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4c5eec4-6f72-4f8c-b30d-1e936e853c19", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
99092025-04-30T06:26:41.953ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
99102025-04-30T06:26:41.959ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99112025-04-30T06:26:41.959ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99122025-04-30T06:26:41.959ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99132025-04-30T06:26:41.960ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99142025-04-30T06:26:41.961ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
99152025-04-30T06:26:41.961ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
99162025-04-30T06:26:41.965ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
99172025-04-30T06:26:41.966ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
99182025-04-30T06:26:41.976ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
99192025-04-30T06:26:41.977ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
99202025-04-30T06:26:41.977ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99212025-04-30T06:26:41.984ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
99222025-04-30T06:26:41.984ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
99232025-04-30T06:26:41.984ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99242025-04-30T06:26:41.984ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99252025-04-30T06:26:41.984ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99262025-04-30T06:26:41.985ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99272025-04-30T06:26:41.991ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
99282025-04-30T06:26:41.992ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
99292025-04-30T06:26:41.992ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99302025-04-30T06:26:41.997ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
99312025-04-30T06:26:41.997ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99322025-04-30T06:26:41.997ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99332025-04-30T06:26:41.998ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99342025-04-30T06:26:42.060ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
99352025-04-30T06:26:42.060ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
99362025-04-30T06:26:42.061ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:63149 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
99372025-04-30T06:26:42.062ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99382025-04-30T06:26:42.062ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99392025-04-30T06:26:42.066ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:63149 req_id = 5fc4c2bc-b64c-4ee8-bb9e-807e8f1e7228 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
99402025-04-30T06:26:42.067ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
99412025-04-30T06:26:42.067ZINFOtest_disk_expunge (dropshot): request completed latency_us = 3153 local_addr = [::1]:44559 method = GET remote_addr = [::1]:63149 req_id = 5fc4c2bc-b64c-4ee8-bb9e-807e8f1e7228 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
99422025-04-30T06:26:42.069ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99432025-04-30T06:26:42.070ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99442025-04-30T06:26:42.070ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99452025-04-30T06:26:42.070ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99462025-04-30T06:26:42.070ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fc4c2bc-b64c-4ee8-bb9e-807e8f1e7228", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
99472025-04-30T06:26:42.072ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
99482025-04-30T06:26:42.072ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
99492025-04-30T06:26:42.072ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
99502025-04-30T06:26:42.080ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
99512025-04-30T06:26:42.080ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99522025-04-30T06:26:42.080ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99532025-04-30T06:26:42.080ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99542025-04-30T06:26:42.082ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99552025-04-30T06:26:42.104ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99562025-04-30T06:26:42.110ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99572025-04-30T06:26:42.111ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99582025-04-30T06:26:42.111ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99592025-04-30T06:26:42.112ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99602025-04-30T06:26:42.153ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
99612025-04-30T06:26:42.173ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
99622025-04-30T06:26:42.173ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
99632025-04-30T06:26:42.173ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 584.548204ms iteration = 34 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
99642025-04-30T06:26:42.173ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 35 reason = Signaled
99652025-04-30T06:26:42.173ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
99662025-04-30T06:26:42.173ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99672025-04-30T06:26:42.173ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99682025-04-30T06:26:42.173ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99692025-04-30T06:26:42.174ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99702025-04-30T06:26:42.178ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99712025-04-30T06:26:42.182ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
99722025-04-30T06:26:42.182ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99732025-04-30T06:26:42.182ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99742025-04-30T06:26:42.183ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99752025-04-30T06:26:42.187ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
99762025-04-30T06:26:42.187ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
99772025-04-30T06:26:42.188ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5c eb 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 \\............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64029 req_id = 29ee61a0-71c7-46c8-bd80-4243ff926bea
99782025-04-30T06:26:42.188ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23787,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64029 req_id = 29ee61a0-71c7-46c8-bd80-4243ff926bea
99792025-04-30T06:26:42.189ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
99802025-04-30T06:26:42.189ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99812025-04-30T06:26:42.189ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64029 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 29ee61a0-71c7-46c8-bd80-4243ff926bea
99822025-04-30T06:26:42.190ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3266, tv_nsec: 989207971 } })
99832025-04-30T06:26:42.190ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
99842025-04-30T06:26:42.190ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2f cc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e /...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59533 req_id = b8f30580-8862-4c6a-ab3a-7b6f873e176a
99852025-04-30T06:26:42.191ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12236,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59533 req_id = b8f30580-8862-4c6a-ab3a-7b6f873e176a
99862025-04-30T06:26:42.192ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
99872025-04-30T06:26:42.192ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59533 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b8f30580-8862-4c6a-ab3a-7b6f873e176a
99882025-04-30T06:26:42.282ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
99892025-04-30T06:26:42.282ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99902025-04-30T06:26:42.282ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99912025-04-30T06:26:42.282ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99922025-04-30T06:26:42.282ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
99932025-04-30T06:26:42.284ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99942025-04-30T06:26:42.284ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b6 5e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .^..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58033 req_id = 412c4c98-d025-484f-9b76-afb47d74d023
99952025-04-30T06:26:42.284ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46686,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58033 req_id = 412c4c98-d025-484f-9b76-afb47d74d023
99962025-04-30T06:26:42.284ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
99972025-04-30T06:26:42.284ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 46686, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58033 req_id = 412c4c98-d025-484f-9b76-afb47d74d023
99982025-04-30T06:26:42.285ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
99992025-04-30T06:26:42.285ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
100002025-04-30T06:26:42.285ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
100012025-04-30T06:26:42.287ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100022025-04-30T06:26:42.295ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100032025-04-30T06:26:42.295ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100042025-04-30T06:26:42.295ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
100052025-04-30T06:26:42.296ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100062025-04-30T06:26:42.303ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b13631c4-fb4c-469b-ae74-d5bc7f0b79dd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100072025-04-30T06:26:42.303ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
100082025-04-30T06:26:42.311ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100092025-04-30T06:26:42.312ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
100102025-04-30T06:26:42.313ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100112025-04-30T06:26:42.314ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0be7b42-e918-4c13-8598-045cf92ddabe", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100122025-04-30T06:26:42.314ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
100132025-04-30T06:26:42.317ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7990462-13db-4649-bf18-7c4f3fbdcbc6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100142025-04-30T06:26:42.318ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
100152025-04-30T06:26:42.324ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c517c0f6-5348-4a2b-8437-cb16c460ad13", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100162025-04-30T06:26:42.324ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
100172025-04-30T06:26:42.327ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d1c5cfe-d1fa-435b-9633-0c15b76d2baa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100182025-04-30T06:26:42.327ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
100192025-04-30T06:26:42.329ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "561cb62d-f8db-44f6-b25f-43e4f990c82c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100202025-04-30T06:26:42.331ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
100212025-04-30T06:26:42.331ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5104741-adaf-4e3c-943b-6b358cfc693e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100222025-04-30T06:26:42.331ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
100232025-04-30T06:26:42.332ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f04bae29-567e-46f9-b4a9-46f7709e91ac", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100242025-04-30T06:26:42.333ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
100252025-04-30T06:26:42.334ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10956eba-d53a-4a0e-8646-1741c85638c6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100262025-04-30T06:26:42.334ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
100272025-04-30T06:26:42.336ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33981818-2d3d-4339-9740-4e9de46a6dce", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100282025-04-30T06:26:42.336ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
100292025-04-30T06:26:42.337ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c7ff291-01ce-4208-b8f0-45dfac2972c1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100302025-04-30T06:26:42.337ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
100312025-04-30T06:26:42.339ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14ef70ef-ce08-4a5e-ba15-c965ec958592", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100322025-04-30T06:26:42.339ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
100332025-04-30T06:26:42.344ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c633bf9d-c554-4259-94ca-e0e1112e35d3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100342025-04-30T06:26:42.344ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
100352025-04-30T06:26:42.348ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd77e49e-c305-43e5-a2ba-75f99195b8b6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100362025-04-30T06:26:42.349ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
100372025-04-30T06:26:42.358ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
100382025-04-30T06:26:42.359ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a11ca50-184d-4262-88c9-f0d5aecf4adb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100392025-04-30T06:26:42.359ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
100402025-04-30T06:26:42.363ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85ffb137-f61d-4da4-839e-6922bfc8f860", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100412025-04-30T06:26:42.363ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
100422025-04-30T06:26:42.367ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dd8dd8b-4de9-445b-8e33-34391d0bc0b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100432025-04-30T06:26:42.367ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
100442025-04-30T06:26:42.371ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad9ac52a-bd39-42c9-8bb8-a62d103ea329", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100452025-04-30T06:26:42.371ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
100462025-04-30T06:26:42.378ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e910998-a0e7-4763-9647-049a9da8c036", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100472025-04-30T06:26:42.378ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
100482025-04-30T06:26:42.379ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69991148-95cd-4b8d-b13c-8daae0df8fef", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100492025-04-30T06:26:42.379ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
100502025-04-30T06:26:42.380ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f58c5a81-da81-409d-b1a8-ef8bdade8a97", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100512025-04-30T06:26:42.380ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
100522025-04-30T06:26:42.381ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97a71e38-377c-4560-8c6a-e17d507aef1f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100532025-04-30T06:26:42.381ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
100542025-04-30T06:26:42.382ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e103ad5c-a653-455d-ba06-650b1e60e50f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100552025-04-30T06:26:42.382ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
100562025-04-30T06:26:42.383ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fb3742a-2c7b-479a-b31e-d96be9f80bd9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100572025-04-30T06:26:42.394ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
100582025-04-30T06:26:42.394ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ee1d223-aec4-4ace-99f9-6586b187670a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:41 GMT"} })
100592025-04-30T06:26:42.394ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
100602025-04-30T06:26:42.394ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f85017a7-1eb6-4ed3-84ad-7adf5a7d2a32", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100612025-04-30T06:26:42.394ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
100622025-04-30T06:26:42.394ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64f1a86e-c07f-4a5d-9f23-2f108d575742", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100632025-04-30T06:26:42.394ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
100642025-04-30T06:26:42.394ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc3d5a5f-126f-4707-8a59-ef2c1ebe6870", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100652025-04-30T06:26:42.394ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
100662025-04-30T06:26:42.394ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90f3addc-8240-4588-8eef-6a5400919aad", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100672025-04-30T06:26:42.394ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
100682025-04-30T06:26:42.398ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91487246-145f-42c6-9822-19c43e00cca7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100692025-04-30T06:26:42.398ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
100702025-04-30T06:26:42.399ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b17aa7e-f3e6-4277-9a35-7b75e47e0dfa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100712025-04-30T06:26:42.399ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
100722025-04-30T06:26:42.400ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4975182-e6be-48ea-a6b2-cdee41d4d253", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100732025-04-30T06:26:42.400ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
100742025-04-30T06:26:42.401ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d4d5c0c-a1d1-4dde-b6b9-a56117a33642", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100752025-04-30T06:26:42.401ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
100762025-04-30T06:26:42.403ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "670fd833-98d2-4667-ad7f-903765f736e5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100772025-04-30T06:26:42.403ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
100782025-04-30T06:26:42.405ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8c19be2-8067-4d9a-b090-dab1ce3acf85", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100792025-04-30T06:26:42.405ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
100802025-04-30T06:26:42.408ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c99deeb1-3032-45f8-b976-d8e8c34df4e8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100812025-04-30T06:26:42.408ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
100822025-04-30T06:26:42.408ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100832025-04-30T06:26:42.409ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100842025-04-30T06:26:42.409ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100852025-04-30T06:26:42.409ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
100862025-04-30T06:26:42.410ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100872025-04-30T06:26:42.420ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100882025-04-30T06:26:42.431ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100892025-04-30T06:26:42.431ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100902025-04-30T06:26:42.431ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
100912025-04-30T06:26:42.433ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100922025-04-30T06:26:42.443ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100932025-04-30T06:26:42.444ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
100942025-04-30T06:26:42.446ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100952025-04-30T06:26:42.495ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "faa1cbbf-c147-4b8c-9426-6ababf7c68d6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100962025-04-30T06:26:42.496ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
100972025-04-30T06:26:42.499ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eea500ce-4c37-418f-bd9b-50fcc19d9b61", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
100982025-04-30T06:26:42.500ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
100992025-04-30T06:26:42.502ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb03433c-2cd8-483e-bfe5-6f4526c15520", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
101002025-04-30T06:26:42.503ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
101012025-04-30T06:26:42.505ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27d4ac71-382d-4588-9130-b9785339bb04", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
101022025-04-30T06:26:42.505ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
101032025-04-30T06:26:42.508ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f1391bc-8bb0-4749-a573-c8657ebe6db1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
101042025-04-30T06:26:42.508ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
101052025-04-30T06:26:42.510ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71ad5467-cb94-4215-a5a8-3dc11e644ef4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
101062025-04-30T06:26:42.511ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
101072025-04-30T06:26:42.515ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b497312b-c237-48a5-87ce-c95b0d797b5a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
101082025-04-30T06:26:42.515ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
101092025-04-30T06:26:42.517ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a9cf70d-27a2-4b3c-9a1e-03bb3be2efcf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
101102025-04-30T06:26:42.518ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
101112025-04-30T06:26:42.520ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc913f77-5de6-44b1-9ba2-b1df613f5dd1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
101122025-04-30T06:26:42.520ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101132025-04-30T06:26:42.521ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101142025-04-30T06:26:42.521ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101152025-04-30T06:26:42.521ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101162025-04-30T06:26:42.522ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101172025-04-30T06:26:42.537ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101182025-04-30T06:26:42.544ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101192025-04-30T06:26:42.545ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101202025-04-30T06:26:42.545ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101212025-04-30T06:26:42.545ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101222025-04-30T06:26:42.545ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101232025-04-30T06:26:42.546ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101242025-04-30T06:26:42.547ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
101252025-04-30T06:26:42.550ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101262025-04-30T06:26:42.557ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101272025-04-30T06:26:42.557ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101282025-04-30T06:26:42.557ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101292025-04-30T06:26:42.558ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101302025-04-30T06:26:42.664ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
101312025-04-30T06:26:42.664ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
101322025-04-30T06:26:42.665ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101332025-04-30T06:26:42.665ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101342025-04-30T06:26:42.665ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101352025-04-30T06:26:42.666ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
101362025-04-30T06:26:42.667ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101372025-04-30T06:26:42.668ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:50216 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
101382025-04-30T06:26:42.668ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:50216 req_id = 0e35cfce-99d5-4d4f-8381-37ce7e8ac682 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
101392025-04-30T06:26:42.669ZINFOtest_disk_expunge (dropshot): request completed latency_us = 529 local_addr = [::1]:44559 method = GET remote_addr = [::1]:50216 req_id = 0e35cfce-99d5-4d4f-8381-37ce7e8ac682 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
101402025-04-30T06:26:42.670ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e35cfce-99d5-4d4f-8381-37ce7e8ac682", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
101412025-04-30T06:26:42.673ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101422025-04-30T06:26:42.673ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101432025-04-30T06:26:42.673ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
101442025-04-30T06:26:42.675ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101452025-04-30T06:26:42.675ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101462025-04-30T06:26:42.675ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101472025-04-30T06:26:42.675ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101482025-04-30T06:26:42.676ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101492025-04-30T06:26:42.677ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101502025-04-30T06:26:42.687ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
101512025-04-30T06:26:42.687ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101522025-04-30T06:26:42.687ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
101532025-04-30T06:26:42.687ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101542025-04-30T06:26:42.690ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101552025-04-30T06:26:42.698ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101562025-04-30T06:26:42.698ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101572025-04-30T06:26:42.698ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101582025-04-30T06:26:42.698ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101592025-04-30T06:26:42.701ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101602025-04-30T06:26:42.701ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
101612025-04-30T06:26:42.702ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101622025-04-30T06:26:42.731ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
101632025-04-30T06:26:42.732ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
101642025-04-30T06:26:42.732ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 558.726706ms iteration = 35 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
101652025-04-30T06:26:42.732ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 36 reason = Signaled
101662025-04-30T06:26:42.732ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101672025-04-30T06:26:42.732ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101682025-04-30T06:26:42.732ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101692025-04-30T06:26:42.732ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101702025-04-30T06:26:42.733ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101712025-04-30T06:26:42.735ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101722025-04-30T06:26:42.740ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101732025-04-30T06:26:42.740ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101742025-04-30T06:26:42.740ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101752025-04-30T06:26:42.741ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101762025-04-30T06:26:42.745ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
101772025-04-30T06:26:42.747ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
101782025-04-30T06:26:42.747ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
101792025-04-30T06:26:42.748ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f2 7b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .{..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54179 req_id = d4311aa7-4618-461c-b4a7-c0c1ba5cb07b
101802025-04-30T06:26:42.748ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62075,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54179 req_id = d4311aa7-4618-461c-b4a7-c0c1ba5cb07b
101812025-04-30T06:26:42.748ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
101822025-04-30T06:26:42.749ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101832025-04-30T06:26:42.749ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54179 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d4311aa7-4618-461c-b4a7-c0c1ba5cb07b
101842025-04-30T06:26:42.749ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3267, tv_nsec: 548433201 } })
101852025-04-30T06:26:42.749ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
101862025-04-30T06:26:42.749ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c1 7d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .}..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58811 req_id = 7ffb31ed-7cbb-45e2-9a9d-a75e26d621af
101872025-04-30T06:26:42.750ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49533,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58811 req_id = 7ffb31ed-7cbb-45e2-9a9d-a75e26d621af
101882025-04-30T06:26:42.750ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101892025-04-30T06:26:42.750ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58811 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7ffb31ed-7cbb-45e2-9a9d-a75e26d621af
101902025-04-30T06:26:42.820ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
101912025-04-30T06:26:42.821ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101922025-04-30T06:26:42.821ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101932025-04-30T06:26:42.821ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101942025-04-30T06:26:42.821ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
101952025-04-30T06:26:42.822ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101962025-04-30T06:26:42.823ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e4 42 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .B..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58450 req_id = faa60082-275d-42a7-bacb-9afa5f3e2fee
101972025-04-30T06:26:42.823ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58434,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58450 req_id = faa60082-275d-42a7-bacb-9afa5f3e2fee
101982025-04-30T06:26:42.824ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
101992025-04-30T06:26:42.824ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 58434, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58450 req_id = faa60082-275d-42a7-bacb-9afa5f3e2fee
102002025-04-30T06:26:42.824ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
102012025-04-30T06:26:42.824ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
102022025-04-30T06:26:42.824ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
102032025-04-30T06:26:42.826ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102042025-04-30T06:26:42.835ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102052025-04-30T06:26:42.835ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102062025-04-30T06:26:42.835ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
102072025-04-30T06:26:42.836ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102082025-04-30T06:26:42.839ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e628abe-c79e-4afe-8720-c83c82750f4a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102092025-04-30T06:26:42.839ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
102102025-04-30T06:26:42.843ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102112025-04-30T06:26:42.843ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
102122025-04-30T06:26:42.844ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102132025-04-30T06:26:42.846ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b62999b0-e992-45c5-9711-fb8b7b8bf85e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102142025-04-30T06:26:42.846ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
102152025-04-30T06:26:42.852ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4694fc34-2128-4526-807e-37d0db72038e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102162025-04-30T06:26:42.852ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
102172025-04-30T06:26:42.855ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1138e20-2485-4fc0-a5b1-fc7129760c34", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102182025-04-30T06:26:42.855ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
102192025-04-30T06:26:42.857ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fa8cf16-7046-4b52-b04d-624e949dfecc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102202025-04-30T06:26:42.858ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
102212025-04-30T06:26:42.867ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56f116df-1033-40e8-ae30-1519f834e572", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102222025-04-30T06:26:42.868ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
102232025-04-30T06:26:42.870ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d45d1b9e-5b3c-49f2-96ac-1903dbb03956", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102242025-04-30T06:26:42.870ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
102252025-04-30T06:26:42.873ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9261eef5-e43c-42a5-9899-a3aee1373200", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102262025-04-30T06:26:42.874ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
102272025-04-30T06:26:42.885ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
102282025-04-30T06:26:42.886ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
102292025-04-30T06:26:42.887ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
102302025-04-30T06:26:42.887ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = c35f2f87-2c7a-45c2-b46e-fb72e1288232 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
102312025-04-30T06:26:42.888ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = c35f2f87-2c7a-45c2-b46e-fb72e1288232 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
102322025-04-30T06:26:42.888ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = c35f2f87-2c7a-45c2-b46e-fb72e1288232 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
102332025-04-30T06:26:42.889ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = c35f2f87-2c7a-45c2-b46e-fb72e1288232 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
102342025-04-30T06:26:42.891ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75649597-fd89-49e8-8eb9-634399eaf325", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102352025-04-30T06:26:42.891ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
102362025-04-30T06:26:42.894ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 6198 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = c35f2f87-2c7a-45c2-b46e-fb72e1288232 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
102372025-04-30T06:26:42.894ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "c35f2f87-2c7a-45c2-b46e-fb72e1288232", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102382025-04-30T06:26:42.895ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
102392025-04-30T06:26:42.901ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4b6f2a0-f79b-4075-a09b-08f0fccd07e9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102402025-04-30T06:26:42.902ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
102412025-04-30T06:26:42.903ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f7985ad7-ecaf-41cb-815d-0df35330b491 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
102422025-04-30T06:26:42.903ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f7985ad7-ecaf-41cb-815d-0df35330b491 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
102432025-04-30T06:26:42.903ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f7985ad7-ecaf-41cb-815d-0df35330b491 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
102442025-04-30T06:26:42.905ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f7985ad7-ecaf-41cb-815d-0df35330b491 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
102452025-04-30T06:26:42.908ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80d1ff88-61f5-4bd5-98d0-dcfa5d3e96d6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102462025-04-30T06:26:42.908ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
102472025-04-30T06:26:42.909ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 6475 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f7985ad7-ecaf-41cb-815d-0df35330b491 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
102482025-04-30T06:26:42.910ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7985ad7-ecaf-41cb-815d-0df35330b491", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102492025-04-30T06:26:42.910ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
102502025-04-30T06:26:42.910ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102512025-04-30T06:26:42.910ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
102522025-04-30T06:26:42.913ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a4302b3-3702-4a47-89e8-dc79eafe8eea", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102532025-04-30T06:26:42.914ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
102542025-04-30T06:26:42.919ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be6dc53c-25c7-444b-bf4a-106fb29a51c0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102552025-04-30T06:26:42.919ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
102562025-04-30T06:26:42.921ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81a5f5d4-8012-4296-8df1-7ed7208ee8ae", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102572025-04-30T06:26:42.921ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
102582025-04-30T06:26:42.923ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91b8a334-f811-4562-ae6d-bb3ee59598ec", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102592025-04-30T06:26:42.923ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
102602025-04-30T06:26:42.925ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef047c21-f24a-4518-9903-e753875c9cca", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102612025-04-30T06:26:42.925ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
102622025-04-30T06:26:42.928ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "762be7c8-3107-42d9-8bf6-90f44b415087", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102632025-04-30T06:26:42.928ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
102642025-04-30T06:26:42.931ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f65d9059-27f5-478b-bfe3-bac977485c35", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102652025-04-30T06:26:42.931ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
102662025-04-30T06:26:42.935ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13636a54-b7b6-4c60-86db-9a8b8bdb77a6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102672025-04-30T06:26:42.935ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
102682025-04-30T06:26:42.936ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4a998b5-249a-4601-b062-80d635502ce3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102692025-04-30T06:26:42.936ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
102702025-04-30T06:26:42.938ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102712025-04-30T06:26:42.938ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102722025-04-30T06:26:42.938ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102732025-04-30T06:26:42.938ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
102742025-04-30T06:26:42.939ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102752025-04-30T06:26:42.940ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1a753df-dfb8-425d-91ad-26d1bfe83b4c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102762025-04-30T06:26:42.941ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
102772025-04-30T06:26:42.943ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45ea7ad6-f68a-45f6-a2d2-753c7775ee63", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102782025-04-30T06:26:42.943ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
102792025-04-30T06:26:42.944ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102802025-04-30T06:26:42.954ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102812025-04-30T06:26:42.954ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102822025-04-30T06:26:42.954ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
102832025-04-30T06:26:42.955ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102842025-04-30T06:26:42.957ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
102852025-04-30T06:26:42.957ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
102862025-04-30T06:26:42.957ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
102872025-04-30T06:26:42.958ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
102882025-04-30T06:26:42.958ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6c80e7f-92bf-4381-a002-fa8b90443056", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102892025-04-30T06:26:42.959ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
102902025-04-30T06:26:42.959ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102912025-04-30T06:26:42.959ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
102922025-04-30T06:26:42.960ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102932025-04-30T06:26:42.973ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
102942025-04-30T06:26:42.973ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
102952025-04-30T06:26:42.980ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
102962025-04-30T06:26:42.980ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
102972025-04-30T06:26:42.982ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ceba820-ee3a-42bc-b5d9-e6b5b04222b9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
102982025-04-30T06:26:42.983ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
102992025-04-30T06:26:42.990ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8468fd30-c0b3-4984-abd1-6ba8b670470a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103002025-04-30T06:26:42.994ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
103012025-04-30T06:26:42.997ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceb967c8-91d5-423c-9590-fd6abd279dcf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103022025-04-30T06:26:43.015ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
103032025-04-30T06:26:43.043ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103042025-04-30T06:26:43.044ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103052025-04-30T06:26:43.052ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103062025-04-30T06:26:43.053ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103072025-04-30T06:26:43.053ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103082025-04-30T06:26:43.053ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103092025-04-30T06:26:43.054ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
103102025-04-30T06:26:43.074ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
103112025-04-30T06:26:43.083ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f16842c-8afc-40fc-b3f3-59cd59300f7d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103122025-04-30T06:26:43.083ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
103132025-04-30T06:26:43.095ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "355966d7-ffb0-468f-a52f-58d4ea0f39ca", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103142025-04-30T06:26:43.095ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
103152025-04-30T06:26:43.099ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02fb1851-227d-41d2-91d6-a7f551fba3e0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103162025-04-30T06:26:43.099ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
103172025-04-30T06:26:43.105ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c0587d6-f141-4418-9608-1b56c5ab9911", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103182025-04-30T06:26:43.105ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
103192025-04-30T06:26:43.105ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103202025-04-30T06:26:43.106ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103212025-04-30T06:26:43.106ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103222025-04-30T06:26:43.106ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
103232025-04-30T06:26:43.108ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103242025-04-30T06:26:43.130ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ff13419-b791-48b8-80a6-7f9a33bb2ddc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103252025-04-30T06:26:43.130ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
103262025-04-30T06:26:43.132ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103272025-04-30T06:26:43.140ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
103282025-04-30T06:26:43.141ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103292025-04-30T06:26:43.143ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
103302025-04-30T06:26:43.144ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103312025-04-30T06:26:43.147ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88f022a7-6223-4ed9-9c61-034b05855ef3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103322025-04-30T06:26:43.147ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
103332025-04-30T06:26:43.148ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea78b555-edaf-404d-8ac2-8b42b9d1e77a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103342025-04-30T06:26:43.149ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
103352025-04-30T06:26:43.150ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103362025-04-30T06:26:43.150ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
103372025-04-30T06:26:43.151ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103382025-04-30T06:26:43.151ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45595666-a1b9-4461-a2c4-4da131230c46", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103392025-04-30T06:26:43.151ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
103402025-04-30T06:26:43.155ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81052545-bb85-4603-9a14-ed35023b6c96", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103412025-04-30T06:26:43.155ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
103422025-04-30T06:26:43.158ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33384e6f-4184-41e2-85cf-82e2d74200c7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103432025-04-30T06:26:43.158ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
103442025-04-30T06:26:43.165ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "077b072e-3e2b-41de-901e-2da45249caba", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103452025-04-30T06:26:43.166ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
103462025-04-30T06:26:43.181ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa41919b-cc2c-457f-ba93-9b13ffce69cc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103472025-04-30T06:26:43.181ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
103482025-04-30T06:26:43.195ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42d77c34-9f0d-4c66-a74b-336fcf665275", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103492025-04-30T06:26:43.195ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
103502025-04-30T06:26:43.202ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
103512025-04-30T06:26:43.208ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32c74436-7a52-43c3-a164-07fd8a390d21", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103522025-04-30T06:26:43.208ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
103532025-04-30T06:26:43.210ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c45759ce-c14b-47ef-8dd6-812f8622aa44", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103542025-04-30T06:26:43.210ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
103552025-04-30T06:26:43.211ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58d406f5-fbee-4224-aadf-6c7bfedcb877", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103562025-04-30T06:26:43.211ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
103572025-04-30T06:26:43.213ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d832adfe-c44a-4271-8d0a-b86d2fa92ae2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103582025-04-30T06:26:43.213ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
103592025-04-30T06:26:43.215ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f71c8112-84b5-4733-b86e-c90b38aa723a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103602025-04-30T06:26:43.216ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
103612025-04-30T06:26:43.217ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88843609-c4aa-4fe0-b868-7244215dd453", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103622025-04-30T06:26:43.217ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103632025-04-30T06:26:43.217ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103642025-04-30T06:26:43.217ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103652025-04-30T06:26:43.217ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103662025-04-30T06:26:43.218ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103672025-04-30T06:26:43.221ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103682025-04-30T06:26:43.225ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103692025-04-30T06:26:43.226ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103702025-04-30T06:26:43.226ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103712025-04-30T06:26:43.226ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103722025-04-30T06:26:43.226ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103732025-04-30T06:26:43.226ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103742025-04-30T06:26:43.230ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103752025-04-30T06:26:43.234ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103762025-04-30T06:26:43.235ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103772025-04-30T06:26:43.235ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103782025-04-30T06:26:43.235ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103792025-04-30T06:26:43.292ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
103802025-04-30T06:26:43.292ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
103812025-04-30T06:26:43.293ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103822025-04-30T06:26:43.293ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103832025-04-30T06:26:43.293ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103842025-04-30T06:26:43.293ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
103852025-04-30T06:26:43.295ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103862025-04-30T06:26:43.295ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:62259 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
103872025-04-30T06:26:43.295ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:62259 req_id = 922e6f79-2bb7-4806-b00e-95848f1a2c5c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103882025-04-30T06:26:43.296ZINFOtest_disk_expunge (dropshot): request completed latency_us = 678 local_addr = [::1]:44559 method = GET remote_addr = [::1]:62259 req_id = 922e6f79-2bb7-4806-b00e-95848f1a2c5c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103892025-04-30T06:26:43.296ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "922e6f79-2bb7-4806-b00e-95848f1a2c5c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:42 GMT"} })
103902025-04-30T06:26:43.299ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103912025-04-30T06:26:43.299ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103922025-04-30T06:26:43.299ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
103932025-04-30T06:26:43.302ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
103942025-04-30T06:26:43.302ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103952025-04-30T06:26:43.302ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103962025-04-30T06:26:43.302ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103972025-04-30T06:26:43.303ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103982025-04-30T06:26:43.304ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103992025-04-30T06:26:43.312ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104002025-04-30T06:26:43.312ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104012025-04-30T06:26:43.312ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
104022025-04-30T06:26:43.313ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104032025-04-30T06:26:43.315ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104042025-04-30T06:26:43.324ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
104052025-04-30T06:26:43.324ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104062025-04-30T06:26:43.324ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104072025-04-30T06:26:43.325ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104082025-04-30T06:26:43.329ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104092025-04-30T06:26:43.329ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
104102025-04-30T06:26:43.330ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104112025-04-30T06:26:43.365ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
104122025-04-30T06:26:43.365ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
104132025-04-30T06:26:43.366ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 633.513624ms iteration = 36 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
104142025-04-30T06:26:43.366ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 37 reason = Signaled
104152025-04-30T06:26:43.367ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
104162025-04-30T06:26:43.367ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104172025-04-30T06:26:43.367ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104182025-04-30T06:26:43.367ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104192025-04-30T06:26:43.368ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104202025-04-30T06:26:43.374ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104212025-04-30T06:26:43.390ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
104222025-04-30T06:26:43.390ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104232025-04-30T06:26:43.390ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104242025-04-30T06:26:43.390ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104252025-04-30T06:26:43.396ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
104262025-04-30T06:26:43.396ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
104272025-04-30T06:26:43.396ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
104282025-04-30T06:26:43.398ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 2c 3f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ,?..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62711 req_id = a4befd3f-f343-4b07-9aa1-7879a5c7da03
104292025-04-30T06:26:43.398ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11327,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62711 req_id = a4befd3f-f343-4b07-9aa1-7879a5c7da03
104302025-04-30T06:26:43.398ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
104312025-04-30T06:26:43.399ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
104322025-04-30T06:26:43.399ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62711 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a4befd3f-f343-4b07-9aa1-7879a5c7da03
104332025-04-30T06:26:43.399ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3268, tv_nsec: 198609264 } })
104342025-04-30T06:26:43.399ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
104352025-04-30T06:26:43.400ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 53 3f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e S?..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61167 req_id = 47dea440-67f6-4c19-9107-1cde267eaadd
104362025-04-30T06:26:43.400ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21311,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61167 req_id = 47dea440-67f6-4c19-9107-1cde267eaadd
104372025-04-30T06:26:43.400ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
104382025-04-30T06:26:43.400ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61167 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 47dea440-67f6-4c19-9107-1cde267eaadd
104392025-04-30T06:26:43.478ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
104402025-04-30T06:26:43.478ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104412025-04-30T06:26:43.479ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104422025-04-30T06:26:43.479ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104432025-04-30T06:26:43.479ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
104442025-04-30T06:26:43.480ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104452025-04-30T06:26:43.480ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9f 42 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .B..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53647 req_id = 042ecc1f-6373-4ef1-818c-7685296467aa
104462025-04-30T06:26:43.481ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40770,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53647 req_id = 042ecc1f-6373-4ef1-818c-7685296467aa
104472025-04-30T06:26:43.481ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
104482025-04-30T06:26:43.481ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40770, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53647 req_id = 042ecc1f-6373-4ef1-818c-7685296467aa
104492025-04-30T06:26:43.482ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
104502025-04-30T06:26:43.482ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
104512025-04-30T06:26:43.482ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
104522025-04-30T06:26:43.483ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104532025-04-30T06:26:43.490ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104542025-04-30T06:26:43.490ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104552025-04-30T06:26:43.490ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
104562025-04-30T06:26:43.491ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104572025-04-30T06:26:43.493ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "598fa97d-e678-4f3a-b47f-e135eef0cc03", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104582025-04-30T06:26:43.493ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
104592025-04-30T06:26:43.494ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104602025-04-30T06:26:43.494ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
104612025-04-30T06:26:43.495ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104622025-04-30T06:26:43.496ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfb64781-37e6-4275-9b6f-f904652c4f36", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104632025-04-30T06:26:43.496ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
104642025-04-30T06:26:43.499ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afc0aad1-7611-4137-b572-1bd089068597", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104652025-04-30T06:26:43.499ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
104662025-04-30T06:26:43.501ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97ac9315-1789-4fc3-a336-0f8ec9401206", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104672025-04-30T06:26:43.501ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
104682025-04-30T06:26:43.506ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "91f33dc6-aa5d-4be1-b1b0-555e46e98e9a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104692025-04-30T06:26:43.506ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
104702025-04-30T06:26:43.509ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97b429a3-eb6c-41e1-888b-540f04b161aa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104712025-04-30T06:26:43.510ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
104722025-04-30T06:26:43.513ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2376299-4d18-4610-bc63-ad90017d3da6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104732025-04-30T06:26:43.513ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
104742025-04-30T06:26:43.526ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c0c0cb9-04e3-4cbe-a655-531a86d23406", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104752025-04-30T06:26:43.526ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
104762025-04-30T06:26:43.532ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa6372a7-fe02-416a-8e7e-8e1c54c87c5b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104772025-04-30T06:26:43.532ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
104782025-04-30T06:26:43.536ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e18998c-8dba-46a2-b27a-371a79c38073", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104792025-04-30T06:26:43.537ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
104802025-04-30T06:26:43.539ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa23c34e-4666-4a63-b9c6-fb3667fd1ba1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104812025-04-30T06:26:43.539ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
104822025-04-30T06:26:43.544ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
104832025-04-30T06:26:43.547ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "226b051d-e0d0-4825-8878-f062e344db77", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104842025-04-30T06:26:43.547ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
104852025-04-30T06:26:43.549ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f07670b6-c3e1-4f7f-8809-0847593f8d75", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104862025-04-30T06:26:43.549ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
104872025-04-30T06:26:43.553ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d341405-b5e2-4593-a573-278ca12e4605", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104882025-04-30T06:26:43.553ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
104892025-04-30T06:26:43.558ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d75e546a-3b27-4203-99bc-ccd44f280158", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104902025-04-30T06:26:43.558ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
104912025-04-30T06:26:43.561ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68a9cd8f-a7d8-4751-8619-ab40d776255d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104922025-04-30T06:26:43.561ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
104932025-04-30T06:26:43.564ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2e5fcb2-0e56-4885-bf4f-03bff26afbcb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104942025-04-30T06:26:43.564ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
104952025-04-30T06:26:43.567ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d4587a6-95a6-4566-953d-86afeb097e25", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104962025-04-30T06:26:43.567ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
104972025-04-30T06:26:43.570ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa966e78-0233-4aaf-8ec6-640b7e11d25a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
104982025-04-30T06:26:43.571ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
104992025-04-30T06:26:43.573ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea8f7b72-ae6a-4e0a-85fe-d8c644f5751d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105002025-04-30T06:26:43.574ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
105012025-04-30T06:26:43.576ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "47aae624-8d76-4c24-ba33-d37f92564fb6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105022025-04-30T06:26:43.576ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
105032025-04-30T06:26:43.577ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb0e96fa-f38f-442e-ad83-9bd682bf41bd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105042025-04-30T06:26:43.578ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
105052025-04-30T06:26:43.579ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "342fb41b-0b9b-4066-a97e-99cf6c79f0c2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105062025-04-30T06:26:43.580ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
105072025-04-30T06:26:43.585ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90e04333-ecc9-45ed-923a-d58a5bb306de", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105082025-04-30T06:26:43.586ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
105092025-04-30T06:26:43.595ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105102025-04-30T06:26:43.595ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105112025-04-30T06:26:43.595ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105122025-04-30T06:26:43.595ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
105132025-04-30T06:26:43.596ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105142025-04-30T06:26:43.596ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a49c7536-b704-49fe-b70b-db3d52475519", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105152025-04-30T06:26:43.597ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
105162025-04-30T06:26:43.600ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105172025-04-30T06:26:43.606ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105182025-04-30T06:26:43.606ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105192025-04-30T06:26:43.606ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
105202025-04-30T06:26:43.606ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105212025-04-30T06:26:43.610ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105222025-04-30T06:26:43.610ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
105232025-04-30T06:26:43.611ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105242025-04-30T06:26:43.614ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b800b32-18ae-49a6-99e8-935a52b13c47", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105252025-04-30T06:26:43.614ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
105262025-04-30T06:26:43.622ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1bb098b-9092-4f63-b602-ddd27e397e5c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105272025-04-30T06:26:43.622ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
105282025-04-30T06:26:43.628ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba4f5574-c6bc-4440-b4b3-7afe82cb925c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105292025-04-30T06:26:43.628ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
105302025-04-30T06:26:43.633ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8301f85-f262-4afc-a0a8-f419f00afa46", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105312025-04-30T06:26:43.633ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
105322025-04-30T06:26:43.637ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dcf854e-c18c-403a-8e9a-a2195b6cb9f6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105332025-04-30T06:26:43.637ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
105342025-04-30T06:26:43.639ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "444ad260-5cc1-4f33-a699-28961d98dcf0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105352025-04-30T06:26:43.639ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
105362025-04-30T06:26:43.644ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22d32179-3e1e-43d7-a653-8fb96e13ece6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105372025-04-30T06:26:43.645ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
105382025-04-30T06:26:43.645ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
105392025-04-30T06:26:43.647ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "915488f9-aa9b-40e3-83fc-dd35103d8249", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105402025-04-30T06:26:43.647ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
105412025-04-30T06:26:43.649ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bbb24ce-8460-4844-bf15-2328583a9adb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105422025-04-30T06:26:43.649ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
105432025-04-30T06:26:43.651ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87432647-e667-4490-ab4c-a3a62cc027a4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105442025-04-30T06:26:43.651ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
105452025-04-30T06:26:43.655ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f109547-e97d-4a28-9037-7409a35214ec", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105462025-04-30T06:26:43.655ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
105472025-04-30T06:26:43.658ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cebcecf-11d3-4bb0-8dc9-711e36106e5f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105482025-04-30T06:26:43.659ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
105492025-04-30T06:26:43.664ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7b3daa0-3f06-4fba-bdb6-c7b852221c5c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105502025-04-30T06:26:43.664ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
105512025-04-30T06:26:43.667ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfd8d6d9-68ea-4528-80cd-f84f69e48d20", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105522025-04-30T06:26:43.668ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
105532025-04-30T06:26:43.671ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20b4287b-cc39-4ce6-a178-aa889b16a398", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105542025-04-30T06:26:43.671ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
105552025-04-30T06:26:43.674ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61bedaa5-6325-4a51-a0eb-0ba4c5c2dd74", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105562025-04-30T06:26:43.675ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
105572025-04-30T06:26:43.676ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd7d1c21-b1da-4820-b76a-538e0bba7fa2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105582025-04-30T06:26:43.676ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
105592025-04-30T06:26:43.678ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "460efce4-e70d-4f6d-95bc-b9b5dee3750c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105602025-04-30T06:26:43.678ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
105612025-04-30T06:26:43.679ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03fca9df-d0a8-46a3-9227-f832737b4147", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105622025-04-30T06:26:43.680ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
105632025-04-30T06:26:43.681ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6353c0e7-6f37-4432-ad6e-2bc1f36bf069", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105642025-04-30T06:26:43.682ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105652025-04-30T06:26:43.682ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105662025-04-30T06:26:43.682ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105672025-04-30T06:26:43.682ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105682025-04-30T06:26:43.683ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105692025-04-30T06:26:43.685ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105702025-04-30T06:26:43.689ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105712025-04-30T06:26:43.689ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105722025-04-30T06:26:43.689ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105732025-04-30T06:26:43.689ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105742025-04-30T06:26:43.690ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105752025-04-30T06:26:43.690ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105762025-04-30T06:26:43.693ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105772025-04-30T06:26:43.700ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105782025-04-30T06:26:43.701ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105792025-04-30T06:26:43.701ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105802025-04-30T06:26:43.702ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105812025-04-30T06:26:43.706ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105822025-04-30T06:26:43.706ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105832025-04-30T06:26:43.706ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105842025-04-30T06:26:43.707ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
105852025-04-30T06:26:43.708ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105862025-04-30T06:26:43.780ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
105872025-04-30T06:26:43.780ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
105882025-04-30T06:26:43.781ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:63871 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
105892025-04-30T06:26:43.782ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:63871 req_id = d1c940db-50be-4086-9d13-c3ed3fac9a06 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
105902025-04-30T06:26:43.782ZINFOtest_disk_expunge (dropshot): request completed latency_us = 519 local_addr = [::1]:44559 method = GET remote_addr = [::1]:63871 req_id = d1c940db-50be-4086-9d13-c3ed3fac9a06 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
105912025-04-30T06:26:43.782ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1c940db-50be-4086-9d13-c3ed3fac9a06", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
105922025-04-30T06:26:43.784ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
105932025-04-30T06:26:43.784ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
105942025-04-30T06:26:43.784ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
105952025-04-30T06:26:43.785ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
105962025-04-30T06:26:43.785ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105972025-04-30T06:26:43.785ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105982025-04-30T06:26:43.785ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105992025-04-30T06:26:43.786ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106002025-04-30T06:26:43.786ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106012025-04-30T06:26:43.793ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106022025-04-30T06:26:43.793ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106032025-04-30T06:26:43.793ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
106042025-04-30T06:26:43.794ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106052025-04-30T06:26:43.796ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106062025-04-30T06:26:43.802ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
106072025-04-30T06:26:43.803ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106082025-04-30T06:26:43.803ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106092025-04-30T06:26:43.804ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106102025-04-30T06:26:43.806ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106112025-04-30T06:26:43.806ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
106122025-04-30T06:26:43.806ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106132025-04-30T06:26:43.836ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
106142025-04-30T06:26:43.836ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
106152025-04-30T06:26:43.836ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 470.351023ms iteration = 37 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
106162025-04-30T06:26:43.836ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 38 reason = Signaled
106172025-04-30T06:26:43.836ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
106182025-04-30T06:26:43.836ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106192025-04-30T06:26:43.836ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106202025-04-30T06:26:43.836ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106212025-04-30T06:26:43.837ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106222025-04-30T06:26:43.841ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106232025-04-30T06:26:43.848ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
106242025-04-30T06:26:43.848ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106252025-04-30T06:26:43.848ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106262025-04-30T06:26:43.849ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106272025-04-30T06:26:43.852ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
106282025-04-30T06:26:43.854ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
106292025-04-30T06:26:43.854ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
106302025-04-30T06:26:43.855ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f5 d0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64652 req_id = 3db83df0-91f2-4a66-ae94-97afd7735c43
106312025-04-30T06:26:43.855ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62928,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64652 req_id = 3db83df0-91f2-4a66-ae94-97afd7735c43
106322025-04-30T06:26:43.857ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
106332025-04-30T06:26:43.857ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106342025-04-30T06:26:43.857ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64652 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3db83df0-91f2-4a66-ae94-97afd7735c43
106352025-04-30T06:26:43.857ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3268, tv_nsec: 655720888 } })
106362025-04-30T06:26:43.857ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
106372025-04-30T06:26:43.857ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2a cd 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e *...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50833 req_id = 38574c63-c604-4207-9b2a-8a3df967d70e
106382025-04-30T06:26:43.857ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10957,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50833 req_id = 38574c63-c604-4207-9b2a-8a3df967d70e
106392025-04-30T06:26:43.857ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106402025-04-30T06:26:43.857ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50833 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 38574c63-c604-4207-9b2a-8a3df967d70e
106412025-04-30T06:26:43.923ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
106422025-04-30T06:26:43.923ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106432025-04-30T06:26:43.923ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106442025-04-30T06:26:43.923ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106452025-04-30T06:26:43.923ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
106462025-04-30T06:26:43.924ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106472025-04-30T06:26:43.924ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: da 44 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .D..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55167 req_id = 87172a9f-1892-4c15-922c-18cbd7c9cf2d
106482025-04-30T06:26:43.925ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55876,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55167 req_id = 87172a9f-1892-4c15-922c-18cbd7c9cf2d
106492025-04-30T06:26:43.925ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
106502025-04-30T06:26:43.925ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55876, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55167 req_id = 87172a9f-1892-4c15-922c-18cbd7c9cf2d
106512025-04-30T06:26:43.926ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
106522025-04-30T06:26:43.926ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
106532025-04-30T06:26:43.926ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
106542025-04-30T06:26:43.927ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106552025-04-30T06:26:43.933ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106562025-04-30T06:26:43.933ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106572025-04-30T06:26:43.933ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
106582025-04-30T06:26:43.934ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106592025-04-30T06:26:43.935ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "67f70237-e5b5-4f4f-8c14-a119ef4a3446", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106602025-04-30T06:26:43.936ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
106612025-04-30T06:26:43.936ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106622025-04-30T06:26:43.937ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
106632025-04-30T06:26:43.937ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106642025-04-30T06:26:43.940ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9d16281-db3b-45d0-8cb7-c68d67034b6f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106652025-04-30T06:26:43.940ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
106662025-04-30T06:26:43.944ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "751d7e9b-06ac-4130-9156-7a653d8c9e1b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106672025-04-30T06:26:43.944ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
106682025-04-30T06:26:43.948ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9fa5318-93be-4b92-afb2-089a4ed036cb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106692025-04-30T06:26:43.948ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
106702025-04-30T06:26:43.953ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5290a398-f8bb-4398-91be-1758a5ba029e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106712025-04-30T06:26:43.953ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
106722025-04-30T06:26:43.955ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4982479-0cc7-4f1c-aba8-27482aa0c418", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106732025-04-30T06:26:43.956ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
106742025-04-30T06:26:43.957ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106752025-04-30T06:26:43.957ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106762025-04-30T06:26:43.957ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
106772025-04-30T06:26:43.960ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
106782025-04-30T06:26:43.967ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
106792025-04-30T06:26:43.967ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
106802025-04-30T06:26:43.970ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3aa86cf0-c3f0-40f0-a34b-3f836ab98fb5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106812025-04-30T06:26:43.970ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
106822025-04-30T06:26:43.973ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
106832025-04-30T06:26:43.973ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106842025-04-30T06:26:43.975ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
106852025-04-30T06:26:43.977ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ff53261-8bcc-4b30-a896-fcaff3027035", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106862025-04-30T06:26:43.977ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
106872025-04-30T06:26:43.981ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76d43c14-fe7d-4735-8c02-999f14d8a702", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106882025-04-30T06:26:43.982ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
106892025-04-30T06:26:43.985ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38d414b1-a0a1-43d1-899d-1382c3412f6a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106902025-04-30T06:26:43.985ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
106912025-04-30T06:26:43.989ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96de52c9-458b-4194-a212-359686616948", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106922025-04-30T06:26:43.989ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
106932025-04-30T06:26:43.990ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d5cb1e6-bbf1-44c5-a855-e2afcd893c66", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106942025-04-30T06:26:43.990ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
106952025-04-30T06:26:43.993ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56720cfd-1add-477f-8182-6f970599ffef", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106962025-04-30T06:26:43.993ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
106972025-04-30T06:26:43.997ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ccf0489-1030-465b-8ea0-a7988aea908d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
106982025-04-30T06:26:43.999ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
106992025-04-30T06:26:44.004ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b657f2b9-af62-43ed-a1bb-2853e134fb18", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107002025-04-30T06:26:44.004ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
107012025-04-30T06:26:44.006ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4afdffd-77c1-4f07-958c-46e56c3f7afe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107022025-04-30T06:26:44.006ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
107032025-04-30T06:26:44.010ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0773ae81-5b9a-4cf7-a6d1-a9597b0468bc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107042025-04-30T06:26:44.010ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
107052025-04-30T06:26:44.013ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "973c6328-0d90-4893-9133-6fa41a70de28", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107062025-04-30T06:26:44.013ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
107072025-04-30T06:26:44.017ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8b5bce5-3b16-4654-923d-1962f8c7fbc6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107082025-04-30T06:26:44.017ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
107092025-04-30T06:26:44.028ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
107102025-04-30T06:26:44.028ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107112025-04-30T06:26:44.028ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107122025-04-30T06:26:44.028ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
107132025-04-30T06:26:44.029ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107142025-04-30T06:26:44.047ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3a342dc-a323-4403-af5a-a65344b160fb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107152025-04-30T06:26:44.047ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
107162025-04-30T06:26:44.047ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107172025-04-30T06:26:44.057ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107182025-04-30T06:26:44.057ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107192025-04-30T06:26:44.057ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
107202025-04-30T06:26:44.063ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107212025-04-30T06:26:44.064ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107222025-04-30T06:26:44.065ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107232025-04-30T06:26:44.074ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
107242025-04-30T06:26:44.075ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107252025-04-30T06:26:44.076ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107262025-04-30T06:26:44.076ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107272025-04-30T06:26:44.084ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
107282025-04-30T06:26:44.084ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107292025-04-30T06:26:44.084ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
107302025-04-30T06:26:44.084ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107312025-04-30T06:26:44.084ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df01a876-cee7-4b66-9870-46ce28cedd85", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107322025-04-30T06:26:44.084ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
107332025-04-30T06:26:44.102ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6274a04d-7b8a-40b7-8f4c-ff98ac6cba85", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107342025-04-30T06:26:44.103ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
107352025-04-30T06:26:44.112ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fc1611d-ba5c-4ec5-a42b-1f4208c92d25", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107362025-04-30T06:26:44.112ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
107372025-04-30T06:26:44.115ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "808bfa45-1a79-4fde-bf54-d1e979ab9193", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107382025-04-30T06:26:44.115ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
107392025-04-30T06:26:44.117ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee229953-0788-4add-99a5-26cf1854871e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107402025-04-30T06:26:44.117ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
107412025-04-30T06:26:44.120ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9cb5314-88e7-4c4e-8968-30050f433dfa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107422025-04-30T06:26:44.120ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
107432025-04-30T06:26:44.122ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ad62a1f-babb-4ff4-aeca-ecefdedf1b56", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107442025-04-30T06:26:44.122ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
107452025-04-30T06:26:44.127ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d93d1291-1b25-4fda-9d2a-bccd3ed322a4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107462025-04-30T06:26:44.127ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
107472025-04-30T06:26:44.130ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0e7cd55-7b24-48f7-a60a-88e455c2b075", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107482025-04-30T06:26:44.130ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
107492025-04-30T06:26:44.134ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
107502025-04-30T06:26:44.144ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18698a25-561a-4d79-b7da-abfa327eb716", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107512025-04-30T06:26:44.144ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
107522025-04-30T06:26:44.146ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3f29759-975d-49f9-b350-796352a26d19", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107532025-04-30T06:26:44.146ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
107542025-04-30T06:26:44.151ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2747e201-79e5-4805-b87c-143a2685e425", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107552025-04-30T06:26:44.151ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
107562025-04-30T06:26:44.152ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59bc12c7-e70f-4d67-bf3e-f005740533eb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107572025-04-30T06:26:44.152ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
107582025-04-30T06:26:44.156ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21eda9ec-75ee-4717-8394-a3b36bd030bc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107592025-04-30T06:26:44.157ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
107602025-04-30T06:26:44.167ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd768d9b-a78d-4656-87f1-903b4dbff378", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107612025-04-30T06:26:44.168ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
107622025-04-30T06:26:44.172ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eeabe69-a803-439c-a9cc-51f78d2af07a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107632025-04-30T06:26:44.173ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
107642025-04-30T06:26:44.177ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f5160f1-9024-49fc-aa5e-49fbed191f2f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107652025-04-30T06:26:44.177ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
107662025-04-30T06:26:44.180ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c67c81be-a394-4bd7-9163-e9d40d5a4754", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107672025-04-30T06:26:44.180ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
107682025-04-30T06:26:44.182ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a6ccec7-68e2-4cc9-aa81-2242ca84a755", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107692025-04-30T06:26:44.183ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
107702025-04-30T06:26:44.185ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
107712025-04-30T06:26:44.185ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107722025-04-30T06:26:44.185ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107732025-04-30T06:26:44.185ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
107742025-04-30T06:26:44.185ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107752025-04-30T06:26:44.187ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2848ec6-e208-4eec-88c3-c720f733a4ed", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107762025-04-30T06:26:44.187ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
107772025-04-30T06:26:44.188ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107782025-04-30T06:26:44.199ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107792025-04-30T06:26:44.199ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107802025-04-30T06:26:44.199ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
107812025-04-30T06:26:44.199ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107822025-04-30T06:26:44.202ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107832025-04-30T06:26:44.202ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
107842025-04-30T06:26:44.202ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107852025-04-30T06:26:44.202ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c9cf024-c1f4-40fc-b55a-2b92bcdf4c8c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107862025-04-30T06:26:44.203ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
107872025-04-30T06:26:44.204ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e3a573d-d7f3-45f4-8f18-318f51173f87", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107882025-04-30T06:26:44.204ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
107892025-04-30T06:26:44.207ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "302b9dce-bbcb-4186-ac0c-0e35008caf35", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107902025-04-30T06:26:44.207ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
107912025-04-30T06:26:44.209ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb9b701d-5524-4358-b992-aa1ba594ecae", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107922025-04-30T06:26:44.210ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
107932025-04-30T06:26:44.213ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d63284aa-47eb-40ed-b57b-5477f50c31f0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
107942025-04-30T06:26:44.213ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
107952025-04-30T06:26:44.213ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107962025-04-30T06:26:44.213ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107972025-04-30T06:26:44.213ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107982025-04-30T06:26:44.213ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107992025-04-30T06:26:44.220ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108002025-04-30T06:26:44.233ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
108012025-04-30T06:26:44.233ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
108022025-04-30T06:26:44.233ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108032025-04-30T06:26:44.233ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108042025-04-30T06:26:44.233ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108052025-04-30T06:26:44.234ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108062025-04-30T06:26:44.238ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108072025-04-30T06:26:44.245ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
108082025-04-30T06:26:44.246ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108092025-04-30T06:26:44.246ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108102025-04-30T06:26:44.246ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108112025-04-30T06:26:44.350ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
108122025-04-30T06:26:44.350ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
108132025-04-30T06:26:44.352ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:39910 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
108142025-04-30T06:26:44.358ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:39910 req_id = f3044b1e-2f60-4163-8ab8-ba6b0f678bf0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
108152025-04-30T06:26:44.359ZINFOtest_disk_expunge (dropshot): request completed latency_us = 544 local_addr = [::1]:44559 method = GET remote_addr = [::1]:39910 req_id = f3044b1e-2f60-4163-8ab8-ba6b0f678bf0 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
108162025-04-30T06:26:44.359ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3044b1e-2f60-4163-8ab8-ba6b0f678bf0", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:43 GMT"} })
108172025-04-30T06:26:44.361ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
108182025-04-30T06:26:44.361ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
108192025-04-30T06:26:44.361ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
108202025-04-30T06:26:44.367ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
108212025-04-30T06:26:44.367ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108222025-04-30T06:26:44.367ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108232025-04-30T06:26:44.367ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108242025-04-30T06:26:44.368ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108252025-04-30T06:26:44.368ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
108262025-04-30T06:26:44.376ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108272025-04-30T06:26:44.380ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
108282025-04-30T06:26:44.380ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108292025-04-30T06:26:44.380ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108302025-04-30T06:26:44.381ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108312025-04-30T06:26:44.428ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
108322025-04-30T06:26:44.428ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108332025-04-30T06:26:44.428ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108342025-04-30T06:26:44.428ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
108352025-04-30T06:26:44.432ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108362025-04-30T06:26:44.437ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108372025-04-30T06:26:44.445ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
108382025-04-30T06:26:44.445ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108392025-04-30T06:26:44.445ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
108402025-04-30T06:26:44.446ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108412025-04-30T06:26:44.457ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108422025-04-30T06:26:44.457ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
108432025-04-30T06:26:44.459ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108442025-04-30T06:26:44.459ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
108452025-04-30T06:26:44.459ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
108462025-04-30T06:26:44.459ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 623.472789ms iteration = 38 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
108472025-04-30T06:26:44.459ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 39 reason = Signaled
108482025-04-30T06:26:44.460ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
108492025-04-30T06:26:44.460ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108502025-04-30T06:26:44.460ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108512025-04-30T06:26:44.460ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108522025-04-30T06:26:44.461ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108532025-04-30T06:26:44.464ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108542025-04-30T06:26:44.472ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
108552025-04-30T06:26:44.472ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108562025-04-30T06:26:44.472ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108572025-04-30T06:26:44.475ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108582025-04-30T06:26:44.485ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
108592025-04-30T06:26:44.486ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
108602025-04-30T06:26:44.487ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6f ca 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 o............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57459 req_id = d397d006-1ead-4cf6-9ee0-d97510d5cc0e
108612025-04-30T06:26:44.487ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28618,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57459 req_id = d397d006-1ead-4cf6-9ee0-d97510d5cc0e
108622025-04-30T06:26:44.487ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
108632025-04-30T06:26:44.487ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108642025-04-30T06:26:44.487ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57459 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d397d006-1ead-4cf6-9ee0-d97510d5cc0e
108652025-04-30T06:26:44.487ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3269, tv_nsec: 286829812 } })
108662025-04-30T06:26:44.488ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
108672025-04-30T06:26:44.489ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fe 56 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .V..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53818 req_id = 5b9d8a20-0fb3-43cf-935a-120273017661
108682025-04-30T06:26:44.489ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65110,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53818 req_id = 5b9d8a20-0fb3-43cf-935a-120273017661
108692025-04-30T06:26:44.489ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
108702025-04-30T06:26:44.489ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53818 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5b9d8a20-0fb3-43cf-935a-120273017661
108712025-04-30T06:26:44.575ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
108722025-04-30T06:26:44.576ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ee 8a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54154 req_id = 59811206-4901-43ab-82b5-9d7538689419
108732025-04-30T06:26:44.576ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61066,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54154 req_id = 59811206-4901-43ab-82b5-9d7538689419
108742025-04-30T06:26:44.577ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
108752025-04-30T06:26:44.577ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61066, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54154 req_id = 59811206-4901-43ab-82b5-9d7538689419
108762025-04-30T06:26:44.577ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
108772025-04-30T06:26:44.578ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
108782025-04-30T06:26:44.578ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
108792025-04-30T06:26:44.581ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "96265092-565d-4015-93a3-04633170927f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
108802025-04-30T06:26:44.581ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
108812025-04-30T06:26:44.587ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbce1b4b-2ec5-4ef5-98f7-b11d90ea8fb3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
108822025-04-30T06:26:44.587ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
108832025-04-30T06:26:44.590ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37704858-34c1-49da-a53f-57ffc00bf827", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
108842025-04-30T06:26:44.590ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
108852025-04-30T06:26:44.593ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
108862025-04-30T06:26:44.597ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df061750-d3eb-48d3-8be0-974eafc3d9a5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
108872025-04-30T06:26:44.597ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
108882025-04-30T06:26:44.600ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "276b94b8-be6b-4fd7-9484-93bf7d7161be", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
108892025-04-30T06:26:44.600ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
108902025-04-30T06:26:44.602ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dbdb0d1-9865-47d6-9af7-660fc3594869", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
108912025-04-30T06:26:44.602ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
108922025-04-30T06:26:44.604ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2829dc27-29b2-4310-b336-eda4a6fffff3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
108932025-04-30T06:26:44.604ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
108942025-04-30T06:26:44.606ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fabcf815-8291-4ac3-b94b-2e5751a8d8a0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
108952025-04-30T06:26:44.607ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
108962025-04-30T06:26:44.608ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "727826ee-5476-496c-8aa8-1f97b91eb787", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
108972025-04-30T06:26:44.608ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
108982025-04-30T06:26:44.609ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e112586-a683-45d8-82f7-d867206b2628", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
108992025-04-30T06:26:44.609ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
109002025-04-30T06:26:44.611ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cfe2083-15fc-4897-b427-745b384dd26b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109012025-04-30T06:26:44.611ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
109022025-04-30T06:26:44.612ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfb4a0a7-c570-4ead-85b5-1e421f35d2ed", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109032025-04-30T06:26:44.612ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
109042025-04-30T06:26:44.614ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "020c5ce4-d02d-4247-a42e-3d9ceab5b0f0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109052025-04-30T06:26:44.614ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
109062025-04-30T06:26:44.617ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e645d311-e1d3-4d24-91cb-991cab0d62fb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109072025-04-30T06:26:44.617ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
109082025-04-30T06:26:44.628ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d71cc61-9a9c-42fe-9e0e-f83225193b9b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109092025-04-30T06:26:44.629ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
109102025-04-30T06:26:44.632ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cafca81-9bec-437a-9634-c2a248c74d42", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109112025-04-30T06:26:44.632ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
109122025-04-30T06:26:44.635ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd4c657a-3929-44cb-8888-9883a00d55be", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109132025-04-30T06:26:44.635ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
109142025-04-30T06:26:44.638ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f41f7352-1c8d-45ee-8dc2-2614d43c4a95", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109152025-04-30T06:26:44.638ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
109162025-04-30T06:26:44.644ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "efa6a415-b6fe-4474-a1db-79be7c6b29ad", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109172025-04-30T06:26:44.644ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
109182025-04-30T06:26:44.645ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
109192025-04-30T06:26:44.645ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109202025-04-30T06:26:44.645ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109212025-04-30T06:26:44.645ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
109222025-04-30T06:26:44.646ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109232025-04-30T06:26:44.650ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "047bd0cc-437a-4108-b6fb-a433a6de2b33", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109242025-04-30T06:26:44.651ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
109252025-04-30T06:26:44.651ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109262025-04-30T06:26:44.677ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
109272025-04-30T06:26:44.677ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109282025-04-30T06:26:44.677ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
109292025-04-30T06:26:44.677ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109302025-04-30T06:26:44.678ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7569df74-da0e-42c5-b6c6-8de15a70aa73", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109312025-04-30T06:26:44.678ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
109322025-04-30T06:26:44.680ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80dd6587-401f-4f1d-811c-9df3075971f1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109332025-04-30T06:26:44.680ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
109342025-04-30T06:26:44.682ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109352025-04-30T06:26:44.682ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
109362025-04-30T06:26:44.682ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109372025-04-30T06:26:44.684ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2291add-79f6-4f77-ac59-60f7f0ca60ba", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109382025-04-30T06:26:44.684ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
109392025-04-30T06:26:44.686ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "266bd01e-5eed-4610-b307-49ba80f40f17", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109402025-04-30T06:26:44.686ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
109412025-04-30T06:26:44.689ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8f9abf0-5ca8-4f27-bb92-e8c6b92b3263", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109422025-04-30T06:26:44.689ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
109432025-04-30T06:26:44.692ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "525ff5a0-6ef0-43db-8152-2586cf7a8492", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109442025-04-30T06:26:44.692ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
109452025-04-30T06:26:44.706ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04391d72-e941-4b12-8b81-d3469d40393b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109462025-04-30T06:26:44.706ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
109472025-04-30T06:26:44.711ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a784e06f-2bf5-4fc4-9a8e-f7f54e30fcb7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109482025-04-30T06:26:44.711ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
109492025-04-30T06:26:44.718ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5538932c-31c7-4557-bfa1-01deb6621083", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109502025-04-30T06:26:44.718ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
109512025-04-30T06:26:44.724ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
109522025-04-30T06:26:44.724ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da6e856f-5de1-41d4-a54e-6fdf48b2a343", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109532025-04-30T06:26:44.725ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
109542025-04-30T06:26:44.726ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc42857b-e561-4eac-b784-4dbad06feb69", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109552025-04-30T06:26:44.726ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
109562025-04-30T06:26:44.728ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cff8a540-83b3-4966-988e-9ffc360951b3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109572025-04-30T06:26:44.728ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
109582025-04-30T06:26:44.730ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab36db1a-23f9-447c-bf1a-c2389c800688", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109592025-04-30T06:26:44.731ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
109602025-04-30T06:26:44.732ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c825502f-3c4f-4631-b0e0-5f60a96911e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109612025-04-30T06:26:44.732ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
109622025-04-30T06:26:44.734ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
109632025-04-30T06:26:44.734ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
109642025-04-30T06:26:44.734ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
109652025-04-30T06:26:44.735ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
109662025-04-30T06:26:44.736ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
109672025-04-30T06:26:44.736ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
109682025-04-30T06:26:44.736ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
109692025-04-30T06:26:44.737ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
109702025-04-30T06:26:44.738ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
109712025-04-30T06:26:44.739ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
109722025-04-30T06:26:44.739ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 126.863\u{b5}s iteration = 3 status = {"error":"task disabled"}
109732025-04-30T06:26:44.739ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "baa67340-d623-4848-9af3-51ecee62685e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
109742025-04-30T06:26:44.740ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
109752025-04-30T06:26:44.740ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
109762025-04-30T06:26:44.743ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
109772025-04-30T06:26:44.743ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
109782025-04-30T06:26:44.743ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109792025-04-30T06:26:44.743ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
109802025-04-30T06:26:44.743ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
109812025-04-30T06:26:44.744ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
109822025-04-30T06:26:44.744ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
109832025-04-30T06:26:44.744ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
109842025-04-30T06:26:44.745ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109852025-04-30T06:26:44.745ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
109862025-04-30T06:26:44.745ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
109872025-04-30T06:26:44.745ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
109882025-04-30T06:26:44.746ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
109892025-04-30T06:26:44.747ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
109902025-04-30T06:26:44.748ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
109912025-04-30T06:26:44.749ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
109922025-04-30T06:26:44.751ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
109932025-04-30T06:26:44.756ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 13.632452ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
109942025-04-30T06:26:44.757ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
109952025-04-30T06:26:44.758ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
109962025-04-30T06:26:44.758ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109972025-04-30T06:26:44.758ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
109982025-04-30T06:26:44.758ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
109992025-04-30T06:26:44.758ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
110002025-04-30T06:26:44.758ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
110012025-04-30T06:26:44.758ZINFOtest_disk_expunge (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
110022025-04-30T06:26:44.758ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = instance_updater elapsed = 65.72\u{b5}s iteration = 2 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
110032025-04-30T06:26:44.758ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
110042025-04-30T06:26:44.758ZINFOtest_disk_expunge (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
110052025-04-30T06:26:44.758ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
110062025-04-30T06:26:44.758ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110072025-04-30T06:26:44.758ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
110082025-04-30T06:26:44.758ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110092025-04-30T06:26:44.759ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110102025-04-30T06:26:44.759ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 8a 2c 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .,..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:54369 req_id = f461838c-441e-4c15-93a0-1b0f274796e6
110112025-04-30T06:26:44.759ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35372,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54369 req_id = f461838c-441e-4c15-93a0-1b0f274796e6
110122025-04-30T06:26:44.759ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
110132025-04-30T06:26:44.759ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110142025-04-30T06:26:44.759ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54369 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f461838c-441e-4c15-93a0-1b0f274796e6
110152025-04-30T06:26:44.760ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110162025-04-30T06:26:44.772ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
110172025-04-30T06:26:44.772ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
110182025-04-30T06:26:44.772ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110192025-04-30T06:26:44.774ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110202025-04-30T06:26:44.775ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110212025-04-30T06:26:44.791ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
110222025-04-30T06:26:44.791ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
110232025-04-30T06:26:44.791ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
110242025-04-30T06:26:44.792ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
110252025-04-30T06:26:44.794ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
110262025-04-30T06:26:44.794ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110272025-04-30T06:26:44.794ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110282025-04-30T06:26:44.795ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110292025-04-30T06:26:44.799ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
110302025-04-30T06:26:44.799ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
110312025-04-30T06:26:44.799ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
110322025-04-30T06:26:44.799ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
110332025-04-30T06:26:44.801ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110342025-04-30T06:26:44.811ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
110352025-04-30T06:26:44.811ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
110362025-04-30T06:26:44.811ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110372025-04-30T06:26:44.811ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
110382025-04-30T06:26:44.811ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110392025-04-30T06:26:44.811ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110402025-04-30T06:26:44.812ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
110412025-04-30T06:26:44.812ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110422025-04-30T06:26:44.813ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110432025-04-30T06:26:44.813ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
110442025-04-30T06:26:44.813ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110452025-04-30T06:26:44.813ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110462025-04-30T06:26:44.813ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
110472025-04-30T06:26:44.817ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110482025-04-30T06:26:44.817ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3269, tv_nsec: 616731178 } })
110492025-04-30T06:26:44.817ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
110502025-04-30T06:26:44.818ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 32 97 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 2............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:62424 req_id = 6257d187-c01b-4d74-85e2-ca9e1c89c488
110512025-04-30T06:26:44.818ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12951,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62424 req_id = 6257d187-c01b-4d74-85e2-ca9e1c89c488
110522025-04-30T06:26:44.818ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
110532025-04-30T06:26:44.819ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110542025-04-30T06:26:44.819ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62424 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6257d187-c01b-4d74-85e2-ca9e1c89c488
110552025-04-30T06:26:44.820ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
110562025-04-30T06:26:44.822ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3269, tv_nsec: 621260292 } })
110572025-04-30T06:26:44.822ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
110582025-04-30T06:26:44.823ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110592025-04-30T06:26:44.835ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
110602025-04-30T06:26:44.835ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
110612025-04-30T06:26:44.835ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110622025-04-30T06:26:44.835ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110632025-04-30T06:26:44.836ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
110642025-04-30T06:26:44.836ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110652025-04-30T06:26:44.836ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
110662025-04-30T06:26:44.836ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110672025-04-30T06:26:44.837ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110682025-04-30T06:26:44.838ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e4 c7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54200 req_id = 9727b86e-5822-41a2-a4ee-b3329e116e8b
110692025-04-30T06:26:44.838ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58567,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54200 req_id = 9727b86e-5822-41a2-a4ee-b3329e116e8b
110702025-04-30T06:26:44.838ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110712025-04-30T06:26:44.838ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54200 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9727b86e-5822-41a2-a4ee-b3329e116e8b
110722025-04-30T06:26:44.838ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 08 2e 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:55610 req_id = 8bbcff4e-2a33-4769-886f-e9f6dd0afb3e
110732025-04-30T06:26:44.839ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2094,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55610 req_id = 8bbcff4e-2a33-4769-886f-e9f6dd0afb3e
110742025-04-30T06:26:44.839ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
110752025-04-30T06:26:44.839ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110762025-04-30T06:26:44.839ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55610 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8bbcff4e-2a33-4769-886f-e9f6dd0afb3e
110772025-04-30T06:26:44.840ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
110782025-04-30T06:26:44.913ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
110792025-04-30T06:26:44.916ZDEBGtest_disk_expunge (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
110802025-04-30T06:26:44.918ZDEBGtest_disk_expunge (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
110812025-04-30T06:26:44.918ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
110822025-04-30T06:26:44.920ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
110832025-04-30T06:26:44.921ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3269, tv_nsec: 720693596 } })
110842025-04-30T06:26:44.921ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
110852025-04-30T06:26:44.922ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 4c a8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e L...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52801 req_id = 37d48b2e-9e15-4089-bebf-e946f31051fc
110862025-04-30T06:26:44.922ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19624,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52801 req_id = 37d48b2e-9e15-4089-bebf-e946f31051fc
110872025-04-30T06:26:44.922ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
110882025-04-30T06:26:44.922ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52801 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 37d48b2e-9e15-4089-bebf-e946f31051fc
110892025-04-30T06:26:44.924ZINFOtest_disk_expunge (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
110902025-04-30T06:26:44.925ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 167.131668ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
110912025-04-30T06:26:44.926ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
110922025-04-30T06:26:45.010ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
110932025-04-30T06:26:45.010ZDEBGtest_disk_expunge (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
110942025-04-30T06:26:45.010ZDEBGtest_disk_expunge (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
110952025-04-30T06:26:45.010ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
110962025-04-30T06:26:45.013ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
110972025-04-30T06:26:45.013ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
110982025-04-30T06:26:45.014ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110992025-04-30T06:26:45.018ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111002025-04-30T06:26:45.026ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
111012025-04-30T06:26:45.026ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
111022025-04-30T06:26:45.026ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111032025-04-30T06:26:45.026ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
111042025-04-30T06:26:45.026ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
111052025-04-30T06:26:45.027ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
111062025-04-30T06:26:45.085ZINFOtest_disk_expunge (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
111072025-04-30T06:26:45.085ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:44559/vpc-routes
111082025-04-30T06:26:45.089ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111092025-04-30T06:26:45.098ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111102025-04-30T06:26:45.099ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111112025-04-30T06:26:45.106ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
111122025-04-30T06:26:45.108ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111132025-04-30T06:26:45.113ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:33367 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
111142025-04-30T06:26:45.113ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 6d981052-bbc4-425f-912d-d37de1f49e6c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111152025-04-30T06:26:45.117ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111162025-04-30T06:26:45.117ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111172025-04-30T06:26:45.118ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111182025-04-30T06:26:45.118ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f6 da 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53092 req_id = 82bb0ec5-300f-4207-9aa6-58a6472ffade
111192025-04-30T06:26:45.119ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63194,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53092 req_id = 82bb0ec5-300f-4207-9aa6-58a6472ffade
111202025-04-30T06:26:45.119ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111212025-04-30T06:26:45.119ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53092 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 82bb0ec5-300f-4207-9aa6-58a6472ffade
111222025-04-30T06:26:45.120ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 6d981052-bbc4-425f-912d-d37de1f49e6c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111232025-04-30T06:26:45.120ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 6d981052-bbc4-425f-912d-d37de1f49e6c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111242025-04-30T06:26:45.121ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 6d981052-bbc4-425f-912d-d37de1f49e6c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111252025-04-30T06:26:45.122ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111262025-04-30T06:26:45.123ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
111272025-04-30T06:26:45.124ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
111282025-04-30T06:26:45.124ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111292025-04-30T06:26:45.132ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
111302025-04-30T06:26:45.132ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
111312025-04-30T06:26:45.132ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
111322025-04-30T06:26:45.133ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
111332025-04-30T06:26:45.138ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
111342025-04-30T06:26:45.228ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
111352025-04-30T06:26:45.228ZDEBGtest_disk_expunge (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
111362025-04-30T06:26:45.228ZDEBGtest_disk_expunge (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
111372025-04-30T06:26:45.229ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
111382025-04-30T06:26:45.230ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
111392025-04-30T06:26:45.231ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:33367 req_id = 7c9fd9ce-82c9-4c88-9375-53f865c7eeba sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
111402025-04-30T06:26:45.234ZINFOtest_disk_expunge (dropshot): request completed latency_us = 3043 local_addr = [::1]:44559 method = GET remote_addr = [::1]:33367 req_id = 7c9fd9ce-82c9-4c88-9375-53f865c7eeba response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
111412025-04-30T06:26:45.234ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 120965 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 6d981052-bbc4-425f-912d-d37de1f49e6c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111422025-04-30T06:26:45.235ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111432025-04-30T06:26:45.235ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
111442025-04-30T06:26:45.236ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111452025-04-30T06:26:45.237ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
111462025-04-30T06:26:45.237ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
111472025-04-30T06:26:45.239ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d981052-bbc4-425f-912d-d37de1f49e6c", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
111482025-04-30T06:26:45.239ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
111492025-04-30T06:26:45.239ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111502025-04-30T06:26:45.240ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111512025-04-30T06:26:45.240ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111522025-04-30T06:26:45.240ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111532025-04-30T06:26:45.240ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:44559/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c9fd9ce-82c9-4c88-9375-53f865c7eeba", "content-length": "2", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
111542025-04-30T06:26:45.241ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
111552025-04-30T06:26:45.241ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
111562025-04-30T06:26:45.242ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
111572025-04-30T06:26:45.242ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
111582025-04-30T06:26:45.313ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:44559/v2p
111592025-04-30T06:26:45.319ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:64674 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
111602025-04-30T06:26:45.321ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 5aac6e15-9e12-4eff-892d-f95cdf6915c0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
111612025-04-30T06:26:45.325ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 5aac6e15-9e12-4eff-892d-f95cdf6915c0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
111622025-04-30T06:26:45.325ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 5aac6e15-9e12-4eff-892d-f95cdf6915c0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
111632025-04-30T06:26:45.333ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 5aac6e15-9e12-4eff-892d-f95cdf6915c0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
111642025-04-30T06:26:45.339ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
111652025-04-30T06:26:45.340ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111662025-04-30T06:26:45.340ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:64674 req_id = afc029a9-5345-42e9-aeeb-d1da552b7837 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
111672025-04-30T06:26:45.354ZINFOtest_disk_expunge (dropshot): request completed latency_us = 13173 local_addr = [::1]:44559 method = GET remote_addr = [::1]:64674 req_id = afc029a9-5345-42e9-aeeb-d1da552b7837 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
111682025-04-30T06:26:45.355ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 34681 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 5aac6e15-9e12-4eff-892d-f95cdf6915c0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
111692025-04-30T06:26:45.356ZINFOtest_disk_expunge (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
111702025-04-30T06:26:45.356ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:44559/eip-gateways
111712025-04-30T06:26:45.358ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = PUT remote_addr = [::1]:33367 req_id = 77af66a1-2d60-4c75-b668-ca6102b6f53b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
111722025-04-30T06:26:45.358ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "5aac6e15-9e12-4eff-892d-f95cdf6915c0", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
111732025-04-30T06:26:45.358ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
111742025-04-30T06:26:45.358ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111752025-04-30T06:26:45.358ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
111762025-04-30T06:26:45.358ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:44559/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "afc029a9-5345-42e9-aeeb-d1da552b7837", "content-length": "2", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
111772025-04-30T06:26:45.359ZINFOtest_disk_expunge (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
111782025-04-30T06:26:45.359ZINFOtest_disk_expunge (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
111792025-04-30T06:26:45.359ZINFOtest_disk_expunge (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
111802025-04-30T06:26:45.365ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 619.595322ms iteration = 2 status = {}
111812025-04-30T06:26:45.366ZINFOtest_disk_expunge (dropshot): request completed latency_us = 8361 local_addr = [::1]:44559 method = PUT remote_addr = [::1]:33367 req_id = 77af66a1-2d60-4c75-b668-ca6102b6f53b response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
111822025-04-30T06:26:45.373ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:44559/eip-gateways", status: 204, headers: {"x-request-id": "77af66a1-2d60-4c75-b668-ca6102b6f53b", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
111832025-04-30T06:26:45.377ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 619.186247ms iteration = 2 status = {}
111842025-04-30T06:26:45.378ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
111852025-04-30T06:26:45.378ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
111862025-04-30T06:26:45.380ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
111872025-04-30T06:26:45.389ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
111882025-04-30T06:26:45.389ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111892025-04-30T06:26:45.389ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
111902025-04-30T06:26:45.389ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
111912025-04-30T06:26:45.390ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
111922025-04-30T06:26:45.394ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7904107c-598a-48ab-9c20-6d6ceb4254c1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
111932025-04-30T06:26:45.394ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
111942025-04-30T06:26:45.395ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111952025-04-30T06:26:45.401ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
111962025-04-30T06:26:45.401ZERROtest_disk_expunge (BackgroundTasks): ipv6 addresses for service zone nat not implemented background_task = service_zone_nat_tracker
111972025-04-30T06:26:45.402ZERROtest_disk_expunge (BackgroundTasks): generated config for fewer than the minimum allowed number of dns zones background_task = service_zone_nat_tracker
111982025-04-30T06:26:45.402ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 659.369541ms iteration = 2 status = {"error":"generated config for fewer than the minimum allowed number of dns zones"}
111992025-04-30T06:26:45.430ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
112002025-04-30T06:26:45.434ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05d2d7f0-2504-4da4-9b06-a0d060dea462", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
112012025-04-30T06:26:45.434ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
112022025-04-30T06:26:45.437ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bb99221-bac3-4326-be60-e22699f2df1b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
112032025-04-30T06:26:45.437ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
112042025-04-30T06:26:45.439ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f004f394-4344-43ca-957c-8871930fe284", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
112052025-04-30T06:26:45.439ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
112062025-04-30T06:26:45.442ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9de02cb7-9c1a-4e15-891f-b6db3afb1988", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
112072025-04-30T06:26:45.442ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
112082025-04-30T06:26:45.444ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fbfd014-60cf-4277-b4b9-94351b7358ca", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
112092025-04-30T06:26:45.444ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
112102025-04-30T06:26:45.446ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "850334d9-d257-47c2-8778-c3854b98c84a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
112112025-04-30T06:26:45.446ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
112122025-04-30T06:26:45.447ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15151ad6-56e3-408d-8765-1f90425d9ef4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
112132025-04-30T06:26:45.447ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
112142025-04-30T06:26:45.448ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77f321dc-edc5-4829-b567-cd4f6b071cb7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
112152025-04-30T06:26:45.448ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
112162025-04-30T06:26:45.449ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28f52685-2f92-4f90-ba63-c307c1c80f6d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
112172025-04-30T06:26:45.449ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
112182025-04-30T06:26:45.449ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112192025-04-30T06:26:45.449ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112202025-04-30T06:26:45.449ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112212025-04-30T06:26:45.450ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112222025-04-30T06:26:45.453ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112232025-04-30T06:26:45.457ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
112242025-04-30T06:26:45.457ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
112252025-04-30T06:26:45.457ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112262025-04-30T06:26:45.457ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112272025-04-30T06:26:45.457ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112282025-04-30T06:26:45.457ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112292025-04-30T06:26:45.460ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112302025-04-30T06:26:45.467ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
112312025-04-30T06:26:45.467ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112322025-04-30T06:26:45.467ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112332025-04-30T06:26:45.468ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112342025-04-30T06:26:45.549ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
112352025-04-30T06:26:45.549ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
112362025-04-30T06:26:45.549ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112372025-04-30T06:26:45.549ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112382025-04-30T06:26:45.549ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112392025-04-30T06:26:45.549ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
112402025-04-30T06:26:45.550ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112412025-04-30T06:26:45.550ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:37257 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
112422025-04-30T06:26:45.551ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:37257 req_id = 3121d4af-ac53-4ff3-a6b3-b8e34bd0db9e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
112432025-04-30T06:26:45.551ZINFOtest_disk_expunge (dropshot): request completed latency_us = 555 local_addr = [::1]:44559 method = GET remote_addr = [::1]:37257 req_id = 3121d4af-ac53-4ff3-a6b3-b8e34bd0db9e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
112442025-04-30T06:26:45.552ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3121d4af-ac53-4ff3-a6b3-b8e34bd0db9e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:44 GMT"} })
112452025-04-30T06:26:45.555ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
112462025-04-30T06:26:45.555ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
112472025-04-30T06:26:45.555ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
112482025-04-30T06:26:45.558ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
112492025-04-30T06:26:45.558ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112502025-04-30T06:26:45.558ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112512025-04-30T06:26:45.558ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112522025-04-30T06:26:45.558ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112532025-04-30T06:26:45.559ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112542025-04-30T06:26:45.564ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112552025-04-30T06:26:45.564ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112562025-04-30T06:26:45.564ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
112572025-04-30T06:26:45.565ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112582025-04-30T06:26:45.568ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112592025-04-30T06:26:45.572ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
112602025-04-30T06:26:45.572ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112612025-04-30T06:26:45.572ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112622025-04-30T06:26:45.572ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112632025-04-30T06:26:45.574ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112642025-04-30T06:26:45.574ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
112652025-04-30T06:26:45.575ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112662025-04-30T06:26:45.609ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
112672025-04-30T06:26:45.609ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
112682025-04-30T06:26:45.610ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.150094467s iteration = 39 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
112692025-04-30T06:26:45.610ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 40 reason = Signaled
112702025-04-30T06:26:45.610ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
112712025-04-30T06:26:45.610ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112722025-04-30T06:26:45.610ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112732025-04-30T06:26:45.610ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112742025-04-30T06:26:45.622ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112752025-04-30T06:26:45.628ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112762025-04-30T06:26:45.638ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
112772025-04-30T06:26:45.638ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
112782025-04-30T06:26:45.638ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
112792025-04-30T06:26:45.640ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
112802025-04-30T06:26:45.642ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
112812025-04-30T06:26:45.647ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
112822025-04-30T06:26:45.647ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
112832025-04-30T06:26:45.648ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 39 14 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 9............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51226 req_id = 6e028094-8dc3-4474-a751-cbb9ca0e68f8
112842025-04-30T06:26:45.648ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14612,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51226 req_id = 6e028094-8dc3-4474-a751-cbb9ca0e68f8
112852025-04-30T06:26:45.648ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
112862025-04-30T06:26:45.649ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112872025-04-30T06:26:45.649ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51226 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6e028094-8dc3-4474-a751-cbb9ca0e68f8
112882025-04-30T06:26:45.649ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3270, tv_nsec: 448633349 } })
112892025-04-30T06:26:45.649ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
112902025-04-30T06:26:45.652ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 88 82 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58609 req_id = 33b84bd8-15e5-444e-a843-2da73cfe4136
112912025-04-30T06:26:45.652ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34946,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58609 req_id = 33b84bd8-15e5-444e-a843-2da73cfe4136
112922025-04-30T06:26:45.652ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112932025-04-30T06:26:45.652ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58609 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 33b84bd8-15e5-444e-a843-2da73cfe4136
112942025-04-30T06:26:45.722ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
112952025-04-30T06:26:45.723ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112962025-04-30T06:26:45.723ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112972025-04-30T06:26:45.723ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112982025-04-30T06:26:45.723ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
112992025-04-30T06:26:45.724ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113002025-04-30T06:26:45.725ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b1 6e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .n..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60421 req_id = 3b46d4d2-69b1-4d90-a420-28174a5610e3
113012025-04-30T06:26:45.726ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45422,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60421 req_id = 3b46d4d2-69b1-4d90-a420-28174a5610e3
113022025-04-30T06:26:45.726ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
113032025-04-30T06:26:45.726ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45422, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60421 req_id = 3b46d4d2-69b1-4d90-a420-28174a5610e3
113042025-04-30T06:26:45.726ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
113052025-04-30T06:26:45.727ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
113062025-04-30T06:26:45.727ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
113072025-04-30T06:26:45.728ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113082025-04-30T06:26:45.741ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113092025-04-30T06:26:45.741ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113102025-04-30T06:26:45.741ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
113112025-04-30T06:26:45.743ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113122025-04-30T06:26:45.747ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f631d255-fe47-4515-8adc-4ac738f47a75", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113132025-04-30T06:26:45.747ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
113142025-04-30T06:26:45.749ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113152025-04-30T06:26:45.749ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
113162025-04-30T06:26:45.750ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113172025-04-30T06:26:45.752ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "042cb18e-d0a7-435d-9cde-01f57056f407", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113182025-04-30T06:26:45.753ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
113192025-04-30T06:26:45.769ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31bf3e01-1497-4a7c-9878-a705a8dfa46f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113202025-04-30T06:26:45.770ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
113212025-04-30T06:26:45.773ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac3aa48a-7f7b-4581-99e2-4c674d8a184b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113222025-04-30T06:26:45.774ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
113232025-04-30T06:26:45.778ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bebe5b0-580a-4888-b2cb-1243eeb8eaec", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113242025-04-30T06:26:45.778ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
113252025-04-30T06:26:45.786ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a923f167-5bb3-4dc8-ac80-b5baa24e9c2e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113262025-04-30T06:26:45.786ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
113272025-04-30T06:26:45.793ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "70a1deba-0c6f-4e95-b8ae-5e80fcfd9c0f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113282025-04-30T06:26:45.793ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
113292025-04-30T06:26:45.800ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7fb0eb6-76a2-4761-8464-110bc991e0a7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113302025-04-30T06:26:45.801ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
113312025-04-30T06:26:45.801ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
113322025-04-30T06:26:45.803ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3a626c3-bebc-47fd-aeba-78857165481f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113332025-04-30T06:26:45.803ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
113342025-04-30T06:26:45.804ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "efb157b5-ddbf-49f9-80dd-fcdcd5fb4f92", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113352025-04-30T06:26:45.804ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
113362025-04-30T06:26:45.805ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8690048-bb24-4417-9a14-39177e70247b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113372025-04-30T06:26:45.806ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
113382025-04-30T06:26:45.807ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eaff9fc2-1246-4957-9c5b-69eba296e496", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113392025-04-30T06:26:45.807ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
113402025-04-30T06:26:45.810ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "07d1aff5-a34b-415c-9b04-ac49510f164c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113412025-04-30T06:26:45.810ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
113422025-04-30T06:26:45.821ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2f45767-e597-4d5f-bde0-b3fc121b966f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113432025-04-30T06:26:45.821ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
113442025-04-30T06:26:45.827ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae0a612a-2789-4cb9-86a7-1f64fb1886cc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113452025-04-30T06:26:45.827ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
113462025-04-30T06:26:45.830ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9c80026-7c59-45fa-b643-b820820af361", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113472025-04-30T06:26:45.830ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
113482025-04-30T06:26:45.834ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c42d7703-2937-49f7-83ac-69fce7fbb237", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113492025-04-30T06:26:45.835ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
113502025-04-30T06:26:45.839ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "679a4579-bfc5-4fcf-a23f-37a493213598", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113512025-04-30T06:26:45.839ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
113522025-04-30T06:26:45.843ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6467cd3-a4c2-4f0b-97a9-aade5663a48c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113532025-04-30T06:26:45.843ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
113542025-04-30T06:26:45.845ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "889ca995-32dd-41ff-b5b7-9ece6a84c994", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113552025-04-30T06:26:45.845ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
113562025-04-30T06:26:45.846ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ade4e5f-ab13-446c-8515-9f9fce86bd8b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113572025-04-30T06:26:45.847ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
113582025-04-30T06:26:45.850ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffd2b468-d63b-4a03-a78c-e321521b8c3e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113592025-04-30T06:26:45.851ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
113602025-04-30T06:26:45.851ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113612025-04-30T06:26:45.852ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113622025-04-30T06:26:45.852ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113632025-04-30T06:26:45.852ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
113642025-04-30T06:26:45.853ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113652025-04-30T06:26:45.853ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2042c639-1b6f-41ba-840c-b3d93cc1f69f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113662025-04-30T06:26:45.853ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
113672025-04-30T06:26:45.855ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4c4a22e-85fd-4f24-9c56-63331b316667", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113682025-04-30T06:26:45.855ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
113692025-04-30T06:26:45.856ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113702025-04-30T06:26:45.866ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113712025-04-30T06:26:45.866ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113722025-04-30T06:26:45.866ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
113732025-04-30T06:26:45.866ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113742025-04-30T06:26:45.870ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d017fd18-fb23-4235-9e86-f395937d6e30", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113752025-04-30T06:26:45.870ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
113762025-04-30T06:26:45.870ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113772025-04-30T06:26:45.870ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
113782025-04-30T06:26:45.871ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113792025-04-30T06:26:45.874ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cc2316c-6c78-4d8a-9cf8-2b70e12b8d31", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113802025-04-30T06:26:45.874ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
113812025-04-30T06:26:45.882ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36800bdc-4ee1-487d-946c-025b8e61cf00", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113822025-04-30T06:26:45.883ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
113832025-04-30T06:26:45.885ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f29ae6a-d4a0-4423-bf21-38bc3c0e4efc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113842025-04-30T06:26:45.885ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
113852025-04-30T06:26:45.892ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa1c327c-9e0a-4bed-b3c6-c731344315a6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113862025-04-30T06:26:45.892ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
113872025-04-30T06:26:45.898ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb587e06-7ca2-47da-a42e-411011ad8338", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113882025-04-30T06:26:45.898ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
113892025-04-30T06:26:45.899ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46e6be01-34ea-4bbd-931c-4f3fd533cb57", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113902025-04-30T06:26:45.899ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
113912025-04-30T06:26:45.900ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "145260a2-b5e3-44fa-bead-67a5a6ebc8f2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113922025-04-30T06:26:45.901ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
113932025-04-30T06:26:45.903ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0de94496-df3c-4287-b940-ca622279f2cd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113942025-04-30T06:26:45.903ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
113952025-04-30T06:26:45.911ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "403ef952-46f2-48e7-b58d-f08d1d8bea29", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113962025-04-30T06:26:45.911ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
113972025-04-30T06:26:45.914ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a3a9670-9a45-4d7a-b468-44c32f6f91d5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
113982025-04-30T06:26:45.914ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
113992025-04-30T06:26:45.927ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
114002025-04-30T06:26:45.936ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75b3d983-b649-4569-86ee-1666d0b3941b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
114012025-04-30T06:26:45.936ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
114022025-04-30T06:26:45.939ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4582c714-d763-4742-835b-b988913fd0cc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
114032025-04-30T06:26:45.940ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
114042025-04-30T06:26:45.943ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33e02cb9-42b8-4bdd-b014-c427d9abe3ac", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
114052025-04-30T06:26:45.945ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
114062025-04-30T06:26:45.948ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93c121f6-d78c-4225-b20b-402e624f25b7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
114072025-04-30T06:26:45.948ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
114082025-04-30T06:26:45.950ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49b96232-f876-44e4-9e4a-bf51172b55b3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
114092025-04-30T06:26:45.951ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
114102025-04-30T06:26:45.956ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114112025-04-30T06:26:45.956ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114122025-04-30T06:26:45.957ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
114132025-04-30T06:26:45.958ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
114142025-04-30T06:26:45.959ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47fe1f27-22d9-4f16-b205-633e16ae70b9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
114152025-04-30T06:26:45.959ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
114162025-04-30T06:26:45.961ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75a06ec6-86e7-4ce2-9db2-1e26df963b06", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
114172025-04-30T06:26:45.961ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
114182025-04-30T06:26:45.962ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
114192025-04-30T06:26:45.962ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
114202025-04-30T06:26:45.963ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "381bb091-2e6f-4de3-930b-60ddee76b243", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
114212025-04-30T06:26:45.964ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
114222025-04-30T06:26:45.965ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "23a44f48-a7e5-4612-9f7a-66aca49d4d29", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
114232025-04-30T06:26:45.965ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
114242025-04-30T06:26:45.967ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6acc51f-1f6d-403e-a627-b17aedf7613c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
114252025-04-30T06:26:45.967ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
114262025-04-30T06:26:45.967ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114272025-04-30T06:26:45.967ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114282025-04-30T06:26:45.967ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114292025-04-30T06:26:45.968ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114302025-04-30T06:26:45.975ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
114312025-04-30T06:26:45.975ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114322025-04-30T06:26:45.976ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114332025-04-30T06:26:45.984ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
114342025-04-30T06:26:45.984ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
114352025-04-30T06:26:45.984ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114362025-04-30T06:26:45.984ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114372025-04-30T06:26:45.984ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114382025-04-30T06:26:45.988ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114392025-04-30T06:26:45.988ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114402025-04-30T06:26:45.988ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114412025-04-30T06:26:45.989ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114422025-04-30T06:26:45.989ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
114432025-04-30T06:26:45.990ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114442025-04-30T06:26:45.994ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114452025-04-30T06:26:46.014ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
114462025-04-30T06:26:46.014ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114472025-04-30T06:26:46.014ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114482025-04-30T06:26:46.016ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114492025-04-30T06:26:46.017ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114502025-04-30T06:26:46.040ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114512025-04-30T06:26:46.043ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114522025-04-30T06:26:46.043ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
114532025-04-30T06:26:46.043ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114542025-04-30T06:26:46.054ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114552025-04-30T06:26:46.054ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114562025-04-30T06:26:46.162ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
114572025-04-30T06:26:46.162ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
114582025-04-30T06:26:46.170ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:42987 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
114592025-04-30T06:26:46.173ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
114602025-04-30T06:26:46.173ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114612025-04-30T06:26:46.173ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
114622025-04-30T06:26:46.174ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114632025-04-30T06:26:46.176ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:42987 req_id = c106ca0a-d070-401e-9638-8bdff39db4f8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
114642025-04-30T06:26:46.177ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114652025-04-30T06:26:46.179ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114662025-04-30T06:26:46.179ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114672025-04-30T06:26:46.179ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114682025-04-30T06:26:46.179ZINFOtest_disk_expunge (dropshot): request completed latency_us = 3262 local_addr = [::1]:44559 method = GET remote_addr = [::1]:42987 req_id = c106ca0a-d070-401e-9638-8bdff39db4f8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
114692025-04-30T06:26:46.195ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c106ca0a-d070-401e-9638-8bdff39db4f8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
114702025-04-30T06:26:46.200ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
114712025-04-30T06:26:46.202ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
114722025-04-30T06:26:46.202ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
114732025-04-30T06:26:46.209ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
114742025-04-30T06:26:46.209ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114752025-04-30T06:26:46.209ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114762025-04-30T06:26:46.209ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114772025-04-30T06:26:46.210ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114782025-04-30T06:26:46.220ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114792025-04-30T06:26:46.225ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
114802025-04-30T06:26:46.225ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114812025-04-30T06:26:46.225ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114822025-04-30T06:26:46.226ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114832025-04-30T06:26:46.269ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
114842025-04-30T06:26:46.274ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
114852025-04-30T06:26:46.275ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
114862025-04-30T06:26:46.275ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 665.12594ms iteration = 40 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
114872025-04-30T06:26:46.275ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 41 reason = Signaled
114882025-04-30T06:26:46.275ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
114892025-04-30T06:26:46.275ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114902025-04-30T06:26:46.275ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114912025-04-30T06:26:46.275ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114922025-04-30T06:26:46.276ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114932025-04-30T06:26:46.280ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114942025-04-30T06:26:46.287ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
114952025-04-30T06:26:46.287ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114962025-04-30T06:26:46.288ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114972025-04-30T06:26:46.288ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114982025-04-30T06:26:46.298ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
114992025-04-30T06:26:46.298ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
115002025-04-30T06:26:46.299ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 30 ed 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 0............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56089 req_id = 6c50eefe-1604-4681-895a-a5a34a5b34b2
115012025-04-30T06:26:46.299ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12525,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56089 req_id = 6c50eefe-1604-4681-895a-a5a34a5b34b2
115022025-04-30T06:26:46.300ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
115032025-04-30T06:26:46.300ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115042025-04-30T06:26:46.300ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56089 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6c50eefe-1604-4681-895a-a5a34a5b34b2
115052025-04-30T06:26:46.301ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3271, tv_nsec: 100093801 } })
115062025-04-30T06:26:46.301ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
115072025-04-30T06:26:46.301ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 65 a9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e e...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54662 req_id = bbec4be1-b58e-495c-a41b-b4c74982431e
115082025-04-30T06:26:46.302ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26025,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54662 req_id = bbec4be1-b58e-495c-a41b-b4c74982431e
115092025-04-30T06:26:46.302ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
115102025-04-30T06:26:46.302ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54662 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = bbec4be1-b58e-495c-a41b-b4c74982431e
115112025-04-30T06:26:46.393ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
115122025-04-30T06:26:46.394ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115132025-04-30T06:26:46.394ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115142025-04-30T06:26:46.394ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115152025-04-30T06:26:46.394ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
115162025-04-30T06:26:46.395ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115172025-04-30T06:26:46.397ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 94 fe 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56890 req_id = ecd61586-4833-4119-936c-b94e8b15e74c
115182025-04-30T06:26:46.397ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38142,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56890 req_id = ecd61586-4833-4119-936c-b94e8b15e74c
115192025-04-30T06:26:46.397ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
115202025-04-30T06:26:46.397ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38142, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56890 req_id = ecd61586-4833-4119-936c-b94e8b15e74c
115212025-04-30T06:26:46.398ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
115222025-04-30T06:26:46.398ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
115232025-04-30T06:26:46.398ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
115242025-04-30T06:26:46.401ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115252025-04-30T06:26:46.414ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115262025-04-30T06:26:46.414ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115272025-04-30T06:26:46.414ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
115282025-04-30T06:26:46.415ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115292025-04-30T06:26:46.417ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6039e3cb-33d4-475d-9863-cd6274e8cc69", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115302025-04-30T06:26:46.417ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
115312025-04-30T06:26:46.418ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6001a420-7cce-4aa8-8765-ba709931aaef", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115322025-04-30T06:26:46.419ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
115332025-04-30T06:26:46.420ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115342025-04-30T06:26:46.420ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
115352025-04-30T06:26:46.420ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115362025-04-30T06:26:46.423ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e999d90-759a-46bd-8bb4-c4b702f27f61", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115372025-04-30T06:26:46.423ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
115382025-04-30T06:26:46.424ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99b688f5-ead2-4125-b7f5-974f7c675ee1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115392025-04-30T06:26:46.425ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
115402025-04-30T06:26:46.427ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "697cd9d8-fcd9-4130-b29b-7580247cf5d6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115412025-04-30T06:26:46.427ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
115422025-04-30T06:26:46.431ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "703f175a-75a0-41d7-a98d-0508fef1dd1b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115432025-04-30T06:26:46.431ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
115442025-04-30T06:26:46.436ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8802e91e-bb60-49b9-b98d-9bd526aa2b12", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115452025-04-30T06:26:46.437ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
115462025-04-30T06:26:46.441ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2c3823e-8146-4a99-94ed-880bc2d84a42", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115472025-04-30T06:26:46.441ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
115482025-04-30T06:26:46.444ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1019682-949a-4a64-9dd4-0cb4ff230dd9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115492025-04-30T06:26:46.444ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
115502025-04-30T06:26:46.445ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4307bfb5-1383-45a3-b12a-8e72a5c46166", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115512025-04-30T06:26:46.446ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
115522025-04-30T06:26:46.447ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9629044e-c408-4274-a0fd-b0dc5fd39a12", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115532025-04-30T06:26:46.448ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
115542025-04-30T06:26:46.450ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "509cc0fd-0a61-46a6-8629-636c0657beb2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115552025-04-30T06:26:46.450ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
115562025-04-30T06:26:46.454ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d4a7c7b-1352-4964-be3f-cc0dd1ac17ea", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115572025-04-30T06:26:46.456ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
115582025-04-30T06:26:46.460ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f48c810-c97a-4fe9-8f2a-3e3c2d932563", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115592025-04-30T06:26:46.460ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
115602025-04-30T06:26:46.464ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "233a94bd-c053-4dec-9822-c86678222ac4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115612025-04-30T06:26:46.464ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
115622025-04-30T06:26:46.471ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
115632025-04-30T06:26:46.476ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "777ff1f1-ba34-497f-9035-e445c9e6bcf5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115642025-04-30T06:26:46.476ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
115652025-04-30T06:26:46.479ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bd602f9-b333-44f6-abb6-7fcbf0876fb6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115662025-04-30T06:26:46.479ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
115672025-04-30T06:26:46.481ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "72163352-42a9-4f01-a22f-b3c86f942fc6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115682025-04-30T06:26:46.481ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
115692025-04-30T06:26:46.496ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14b27405-13b6-4301-b459-2ecb673dcb02", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115702025-04-30T06:26:46.496ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
115712025-04-30T06:26:46.498ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4b3367c-6d87-4de7-94bb-5f3c1c28506e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115722025-04-30T06:26:46.499ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
115732025-04-30T06:26:46.501ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "620b095f-c36d-4881-b869-5cbcdbdbbe43", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115742025-04-30T06:26:46.502ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
115752025-04-30T06:26:46.503ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e14bff8f-cf2d-4fb2-9ca6-76c0936e7bb2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115762025-04-30T06:26:46.503ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
115772025-04-30T06:26:46.506ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a80a4b3d-75e2-48b0-918e-797e579ea4b8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115782025-04-30T06:26:46.506ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
115792025-04-30T06:26:46.510ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1e07835-c32e-45ac-9709-075bfc213fd5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115802025-04-30T06:26:46.510ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
115812025-04-30T06:26:46.514ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db32e818-71de-42d0-83df-761cd1189643", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115822025-04-30T06:26:46.514ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
115832025-04-30T06:26:46.517ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c200652-69d0-4ada-b852-17f06b4c5e6e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115842025-04-30T06:26:46.517ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
115852025-04-30T06:26:46.520ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7e32398-dd8a-43bf-8f64-4b36b75864b6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115862025-04-30T06:26:46.520ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
115872025-04-30T06:26:46.522ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115882025-04-30T06:26:46.522ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115892025-04-30T06:26:46.522ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115902025-04-30T06:26:46.522ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
115912025-04-30T06:26:46.523ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115922025-04-30T06:26:46.526ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f31a2f2b-1ae2-4479-8428-36a72dc7c6ee", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
115932025-04-30T06:26:46.533ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
115942025-04-30T06:26:46.535ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115952025-04-30T06:26:46.544ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115962025-04-30T06:26:46.544ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115972025-04-30T06:26:46.544ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
115982025-04-30T06:26:46.545ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115992025-04-30T06:26:46.549ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116002025-04-30T06:26:46.549ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
116012025-04-30T06:26:46.551ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116022025-04-30T06:26:46.553ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cfc6cc5-f2c6-4709-aac4-62d87629456c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116032025-04-30T06:26:46.554ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
116042025-04-30T06:26:46.557ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed178727-8c5e-4ea2-8758-15163c88ef1c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116052025-04-30T06:26:46.558ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
116062025-04-30T06:26:46.559ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96a2bf96-a48c-4216-afc3-df8e39f14420", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116072025-04-30T06:26:46.559ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
116082025-04-30T06:26:46.561ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "926a4920-29a0-4a80-b539-ac0376517282", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116092025-04-30T06:26:46.561ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
116102025-04-30T06:26:46.563ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a0931cb-dfb7-462f-b0cf-d67521ba24ca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116112025-04-30T06:26:46.563ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
116122025-04-30T06:26:46.565ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44cc7df2-158e-43cc-ab1e-f249c89ee564", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116132025-04-30T06:26:46.565ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
116142025-04-30T06:26:46.568ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b39c1093-33d1-4317-aafc-5a09847a7a51", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116152025-04-30T06:26:46.568ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
116162025-04-30T06:26:46.573ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "03b2d353-7648-4233-a64a-1b7c5c2e183b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116172025-04-30T06:26:46.573ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
116182025-04-30T06:26:46.576ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdc3fcba-d138-4008-8ec7-87e846900948", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116192025-04-30T06:26:46.578ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
116202025-04-30T06:26:46.584ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
116212025-04-30T06:26:46.585ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "53db913f-0b09-4cb1-a3cd-f6ace4d5d298", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116222025-04-30T06:26:46.585ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
116232025-04-30T06:26:46.590ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea1a04f6-d41d-4a91-b6d7-a1acd5f0c9e2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116242025-04-30T06:26:46.590ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
116252025-04-30T06:26:46.595ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02ab8bef-f6e9-48b8-88f5-7688271cf35e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116262025-04-30T06:26:46.595ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
116272025-04-30T06:26:46.599ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe468bca-2381-4a62-9770-ad8a0af30a12", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116282025-04-30T06:26:46.599ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
116292025-04-30T06:26:46.601ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ebbbfab-f3b8-4471-bde1-a7dc61c1f5bc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116302025-04-30T06:26:46.601ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
116312025-04-30T06:26:46.603ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd83d47b-eede-4bc5-966c-45133423c9ca", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116322025-04-30T06:26:46.603ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
116332025-04-30T06:26:46.604ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "567084c5-56c2-4aa1-8eac-b8092497febc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116342025-04-30T06:26:46.604ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
116352025-04-30T06:26:46.606ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eecff2ae-230e-473a-8b74-411394fad5eb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:45 GMT"} })
116362025-04-30T06:26:46.606ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
116372025-04-30T06:26:46.607ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116382025-04-30T06:26:46.607ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116392025-04-30T06:26:46.607ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116402025-04-30T06:26:46.608ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116412025-04-30T06:26:46.611ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116422025-04-30T06:26:46.621ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
116432025-04-30T06:26:46.621ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
116442025-04-30T06:26:46.621ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116452025-04-30T06:26:46.621ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116462025-04-30T06:26:46.621ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116472025-04-30T06:26:46.622ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116482025-04-30T06:26:46.625ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116492025-04-30T06:26:46.631ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
116502025-04-30T06:26:46.632ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116512025-04-30T06:26:46.632ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116522025-04-30T06:26:46.636ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116532025-04-30T06:26:46.637ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116542025-04-30T06:26:46.638ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116552025-04-30T06:26:46.639ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116562025-04-30T06:26:46.640ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
116572025-04-30T06:26:46.642ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116582025-04-30T06:26:46.645ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116592025-04-30T06:26:46.656ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116602025-04-30T06:26:46.657ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116612025-04-30T06:26:46.657ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
116622025-04-30T06:26:46.657ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116632025-04-30T06:26:46.745ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
116642025-04-30T06:26:46.745ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
116652025-04-30T06:26:46.746ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:57192 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
116662025-04-30T06:26:46.747ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:57192 req_id = 4fb086c6-b4cc-4aef-97a5-4ea3c6bbe170 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
116672025-04-30T06:26:46.747ZINFOtest_disk_expunge (dropshot): request completed latency_us = 729 local_addr = [::1]:44559 method = GET remote_addr = [::1]:57192 req_id = 4fb086c6-b4cc-4aef-97a5-4ea3c6bbe170 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
116682025-04-30T06:26:46.748ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116692025-04-30T06:26:46.748ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
116702025-04-30T06:26:46.748ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116712025-04-30T06:26:46.749ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fb086c6-b4cc-4aef-97a5-4ea3c6bbe170", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
116722025-04-30T06:26:46.751ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
116732025-04-30T06:26:46.751ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
116742025-04-30T06:26:46.751ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
116752025-04-30T06:26:46.756ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
116762025-04-30T06:26:46.756ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116772025-04-30T06:26:46.756ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116782025-04-30T06:26:46.756ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116792025-04-30T06:26:46.757ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116802025-04-30T06:26:46.762ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116812025-04-30T06:26:46.768ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
116822025-04-30T06:26:46.769ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116832025-04-30T06:26:46.769ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116842025-04-30T06:26:46.770ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116852025-04-30T06:26:46.800ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
116862025-04-30T06:26:46.810ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
116872025-04-30T06:26:46.810ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
116882025-04-30T06:26:46.810ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 535.403707ms iteration = 41 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
116892025-04-30T06:26:46.810ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 42 reason = Signaled
116902025-04-30T06:26:46.810ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
116912025-04-30T06:26:46.810ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116922025-04-30T06:26:46.810ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116932025-04-30T06:26:46.811ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116942025-04-30T06:26:46.813ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116952025-04-30T06:26:46.817ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116962025-04-30T06:26:46.825ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
116972025-04-30T06:26:46.825ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116982025-04-30T06:26:46.825ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116992025-04-30T06:26:46.826ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117002025-04-30T06:26:46.833ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
117012025-04-30T06:26:46.833ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
117022025-04-30T06:26:46.834ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 44 9d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 D............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57514 req_id = b801ccc7-1431-415f-bdb7-c057cda47835
117032025-04-30T06:26:46.834ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17565,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57514 req_id = b801ccc7-1431-415f-bdb7-c057cda47835
117042025-04-30T06:26:46.838ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
117052025-04-30T06:26:46.838ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117062025-04-30T06:26:46.838ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57514 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b801ccc7-1431-415f-bdb7-c057cda47835
117072025-04-30T06:26:46.838ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3271, tv_nsec: 635691555 } })
117082025-04-30T06:26:46.838ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
117092025-04-30T06:26:46.838ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 72 5a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e rZ..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62838 req_id = 19ca7f11-20c1-4c47-921f-12af61118ee0
117102025-04-30T06:26:46.838ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 29274,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62838 req_id = 19ca7f11-20c1-4c47-921f-12af61118ee0
117112025-04-30T06:26:46.838ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117122025-04-30T06:26:46.838ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62838 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 19ca7f11-20c1-4c47-921f-12af61118ee0
117132025-04-30T06:26:46.920ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
117142025-04-30T06:26:46.921ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117152025-04-30T06:26:46.921ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117162025-04-30T06:26:46.921ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117172025-04-30T06:26:46.921ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
117182025-04-30T06:26:46.922ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117192025-04-30T06:26:46.923ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
117202025-04-30T06:26:46.925ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
117212025-04-30T06:26:46.926ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 81727c8d-1c87-4b34-9770-9e26a1ee0c0e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
117222025-04-30T06:26:46.926ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
117232025-04-30T06:26:46.926ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 76 e0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c v............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57758 req_id = 7edab948-6907-41d9-ab43-aa54181f0143
117242025-04-30T06:26:46.926ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30432,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57758 req_id = 7edab948-6907-41d9-ab43-aa54181f0143
117252025-04-30T06:26:46.927ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
117262025-04-30T06:26:46.927ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30432, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57758 req_id = 7edab948-6907-41d9-ab43-aa54181f0143
117272025-04-30T06:26:46.928ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 81727c8d-1c87-4b34-9770-9e26a1ee0c0e resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
117282025-04-30T06:26:46.928ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 81727c8d-1c87-4b34-9770-9e26a1ee0c0e roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
117292025-04-30T06:26:46.928ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 81727c8d-1c87-4b34-9770-9e26a1ee0c0e resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
117302025-04-30T06:26:46.928ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
117312025-04-30T06:26:46.928ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
117322025-04-30T06:26:46.928ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
117332025-04-30T06:26:46.929ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 7e ad 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ~............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52759 req_id = dd423a1e-4f9a-4507-8c50-2c7e4cdfbfe9
117342025-04-30T06:26:46.929ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32429,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52759 req_id = dd423a1e-4f9a-4507-8c50-2c7e4cdfbfe9
117352025-04-30T06:26:46.929ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
117362025-04-30T06:26:46.929ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117372025-04-30T06:26:46.929ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52759 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = dd423a1e-4f9a-4507-8c50-2c7e4cdfbfe9
117382025-04-30T06:26:46.930ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117392025-04-30T06:26:46.935ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
117402025-04-30T06:26:46.935ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117412025-04-30T06:26:46.935ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
117422025-04-30T06:26:46.936ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117432025-04-30T06:26:46.936ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3271, tv_nsec: 735835695 } })
117442025-04-30T06:26:46.937ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
117452025-04-30T06:26:46.939ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 13012 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 81727c8d-1c87-4b34-9770-9e26a1ee0c0e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
117462025-04-30T06:26:46.943ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "81727c8d-1c87-4b34-9770-9e26a1ee0c0e", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
117472025-04-30T06:26:46.944ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
117482025-04-30T06:26:46.950ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117492025-04-30T06:26:46.950ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
117502025-04-30T06:26:46.952ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117512025-04-30T06:26:46.953ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = e339c8b9-a564-49b8-b589-13395d8f3df4 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
117522025-04-30T06:26:46.954ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b1 fe 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57482 req_id = c36048d7-9b45-44b5-9f3a-9ef2b84264f7
117532025-04-30T06:26:46.954ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45566,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57482 req_id = c36048d7-9b45-44b5-9f3a-9ef2b84264f7
117542025-04-30T06:26:46.954ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117552025-04-30T06:26:46.954ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57482 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c36048d7-9b45-44b5-9f3a-9ef2b84264f7
117562025-04-30T06:26:46.955ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = e339c8b9-a564-49b8-b589-13395d8f3df4 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
117572025-04-30T06:26:46.955ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = e339c8b9-a564-49b8-b589-13395d8f3df4 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
117582025-04-30T06:26:46.956ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = e339c8b9-a564-49b8-b589-13395d8f3df4 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
117592025-04-30T06:26:46.959ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
117602025-04-30T06:26:47.047ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
117612025-04-30T06:26:47.047ZDEBGtest_disk_expunge (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
117622025-04-30T06:26:47.047ZDEBGtest_disk_expunge (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
117632025-04-30T06:26:47.047ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
117642025-04-30T06:26:47.050ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
117652025-04-30T06:26:47.050ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117662025-04-30T06:26:47.050ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
117672025-04-30T06:26:47.050ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "23535caf-0de5-4621-8608-be8a92c201b7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
117682025-04-30T06:26:47.050ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
117692025-04-30T06:26:47.051ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
117702025-04-30T06:26:47.052ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
117712025-04-30T06:26:47.056ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
117722025-04-30T06:26:47.056ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
117732025-04-30T06:26:47.057ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117742025-04-30T06:26:47.057ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
117752025-04-30T06:26:47.062ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117762025-04-30T06:26:47.062ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 109132 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = e339c8b9-a564-49b8-b589-13395d8f3df4 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
117772025-04-30T06:26:47.064ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
117782025-04-30T06:26:47.064ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
117792025-04-30T06:26:47.065ZTRACtest_disk_expunge (dropshot): incoming request local_addr = 127.0.0.1:38728 method = GET remote_addr = 127.0.0.1:42469 req_id = a5188eee-fae5-4f67-8c78-97fc8093f04d uri = /78121327-166d-4e33-935c-82ab8635bf33
117802025-04-30T06:26:47.067ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "e339c8b9-a564-49b8-b589-13395d8f3df4", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
117812025-04-30T06:26:47.067ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
117822025-04-30T06:26:47.067ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117832025-04-30T06:26:47.067ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
117842025-04-30T06:26:47.073ZINFOtest_disk_expunge (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 7572 local_addr = 127.0.0.1:38728 method = GET remote_addr = 127.0.0.1:42469 req_id = a5188eee-fae5-4f67-8c78-97fc8093f04d response_code = 200 uri = /78121327-166d-4e33-935c-82ab8635bf33
117852025-04-30T06:26:47.086ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 58 e7 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 X............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52196 req_id = 87d4073f-adb9-4766-b0c3-6aacc50fcd6c
117862025-04-30T06:26:47.088ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22759,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52196 req_id = 87d4073f-adb9-4766-b0c3-6aacc50fcd6c
117872025-04-30T06:26:47.088ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
117882025-04-30T06:26:47.088ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117892025-04-30T06:26:47.088ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52196 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 87d4073f-adb9-4766-b0c3-6aacc50fcd6c
117902025-04-30T06:26:47.089ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
117912025-04-30T06:26:47.089ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117922025-04-30T06:26:47.090ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117932025-04-30T06:26:47.090ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117942025-04-30T06:26:47.090ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
117952025-04-30T06:26:47.100ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
117962025-04-30T06:26:47.100ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
117972025-04-30T06:26:47.100ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
117982025-04-30T06:26:47.104ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3271, tv_nsec: 903476188 } })
117992025-04-30T06:26:47.104ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
118002025-04-30T06:26:47.105ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
118012025-04-30T06:26:47.105ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
118022025-04-30T06:26:47.119ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ad b6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61165 req_id = 5bd68bc4-9b4b-4c94-9140-aab46b1c84db
118032025-04-30T06:26:47.119ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44470,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61165 req_id = 5bd68bc4-9b4b-4c94-9140-aab46b1c84db
118042025-04-30T06:26:47.119ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118052025-04-30T06:26:47.119ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61165 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5bd68bc4-9b4b-4c94-9140-aab46b1c84db
118062025-04-30T06:26:47.121ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
118072025-04-30T06:26:47.189ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
118082025-04-30T06:26:47.189ZDEBGtest_disk_expunge (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
118092025-04-30T06:26:47.190ZDEBGtest_disk_expunge (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
118102025-04-30T06:26:47.190ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
118112025-04-30T06:26:47.193ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
118122025-04-30T06:26:47.196ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
118132025-04-30T06:26:47.203ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec650757-857f-48c7-a233-a52311b050fc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118142025-04-30T06:26:47.203ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
118152025-04-30T06:26:47.233ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
118162025-04-30T06:26:47.234ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: be 53 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .S..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:61873 req_id = e9a023d8-72a6-4ca8-831e-639e0493b4fe
118172025-04-30T06:26:47.234ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48723,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61873 req_id = e9a023d8-72a6-4ca8-831e-639e0493b4fe
118182025-04-30T06:26:47.234ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
118192025-04-30T06:26:47.235ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118202025-04-30T06:26:47.235ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61873 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e9a023d8-72a6-4ca8-831e-639e0493b4fe
118212025-04-30T06:26:47.235ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3272, tv_nsec: 34551076 } })
118222025-04-30T06:26:47.235ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
118232025-04-30T06:26:47.236ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8b b0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55981 req_id = 94476a80-746e-4f10-97f0-81d97e1d3d30
118242025-04-30T06:26:47.236ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35760,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55981 req_id = 94476a80-746e-4f10-97f0-81d97e1d3d30
118252025-04-30T06:26:47.236ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118262025-04-30T06:26:47.236ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55981 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 94476a80-746e-4f10-97f0-81d97e1d3d30
118272025-04-30T06:26:47.238ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
118282025-04-30T06:26:47.297ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
118292025-04-30T06:26:47.297ZDEBGtest_disk_expunge (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
118302025-04-30T06:26:47.297ZDEBGtest_disk_expunge (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
118312025-04-30T06:26:47.298ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
118322025-04-30T06:26:47.301ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
118332025-04-30T06:26:47.301ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "279e77de-f5f7-4a4e-a279-c39095805ca8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118342025-04-30T06:26:47.301ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
118352025-04-30T06:26:47.302ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
118362025-04-30T06:26:47.306ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c5c755d-213c-4da2-a83e-8bece34513d5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118372025-04-30T06:26:47.306ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
118382025-04-30T06:26:47.308ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4da4ea63-4cf7-4602-a3b9-d612d7772b54", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118392025-04-30T06:26:47.308ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
118402025-04-30T06:26:47.311ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59652436-43a3-40b5-987a-c636f6112245", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118412025-04-30T06:26:47.311ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
118422025-04-30T06:26:47.314ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99c09b3a-cec6-43f4-b027-fa2a2596ab6b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118432025-04-30T06:26:47.314ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
118442025-04-30T06:26:47.318ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9074747a-95f6-4455-9740-2dfe20939f7e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118452025-04-30T06:26:47.318ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
118462025-04-30T06:26:47.322ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0693c52-add9-4b59-9ef2-e9ecec670f62", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118472025-04-30T06:26:47.322ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
118482025-04-30T06:26:47.325ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57dc6130-e988-4357-a357-040f6044b178", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118492025-04-30T06:26:47.325ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
118502025-04-30T06:26:47.329ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35cc67b5-77d1-4f9f-963e-4f8c493173ae", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118512025-04-30T06:26:47.329ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
118522025-04-30T06:26:47.330ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf477f1d-729b-4a0d-81b0-518cb5158b15", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118532025-04-30T06:26:47.330ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
118542025-04-30T06:26:47.332ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00c99812-94d3-4c07-b04a-cf98b5885fd7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118552025-04-30T06:26:47.332ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
118562025-04-30T06:26:47.336ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "268d6854-31bb-481b-82f8-f6948e885911", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118572025-04-30T06:26:47.336ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
118582025-04-30T06:26:47.341ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5c394dc-fbcd-4c95-94f8-9f5d45703b59", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118592025-04-30T06:26:47.342ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
118602025-04-30T06:26:47.346ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d13ad36b-c798-4c0d-849c-f0cbff8aea5a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118612025-04-30T06:26:47.347ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
118622025-04-30T06:26:47.349ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36126d5a-e9e4-4acd-a143-d1ad3e6308a0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118632025-04-30T06:26:47.349ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
118642025-04-30T06:26:47.353ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118652025-04-30T06:26:47.353ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118662025-04-30T06:26:47.353ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118672025-04-30T06:26:47.353ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
118682025-04-30T06:26:47.354ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118692025-04-30T06:26:47.357ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "142d56a3-ef98-401f-9112-661085a0d869", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118702025-04-30T06:26:47.357ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
118712025-04-30T06:26:47.359ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118722025-04-30T06:26:47.368ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118732025-04-30T06:26:47.368ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118742025-04-30T06:26:47.368ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
118752025-04-30T06:26:47.369ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118762025-04-30T06:26:47.373ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118772025-04-30T06:26:47.373ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
118782025-04-30T06:26:47.373ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118792025-04-30T06:26:47.375ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c992afb7-e4e4-4a4f-b8a8-fb56059ebc27", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118802025-04-30T06:26:47.375ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
118812025-04-30T06:26:47.378ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c188809-0ec6-4ab4-9eee-31616c645fc2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118822025-04-30T06:26:47.378ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
118832025-04-30T06:26:47.380ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67040fb2-6daf-4a73-abe0-03db4f77ac91", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118842025-04-30T06:26:47.381ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
118852025-04-30T06:26:47.382ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e06b06e-435e-47bf-94ef-aa57257d4ac7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118862025-04-30T06:26:47.382ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
118872025-04-30T06:26:47.383ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69a3a582-cdbb-4d2b-a8bd-8002b94a66c1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118882025-04-30T06:26:47.383ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
118892025-04-30T06:26:47.384ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae9fd245-bc47-4338-b095-fb4fb0734019", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118902025-04-30T06:26:47.384ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
118912025-04-30T06:26:47.386ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc21b45b-ab35-41ca-b739-a11e2aebb927", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118922025-04-30T06:26:47.387ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
118932025-04-30T06:26:47.388ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68555520-c4a9-4df3-96e1-7ae9709ffd43", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118942025-04-30T06:26:47.388ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
118952025-04-30T06:26:47.390ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06936cc9-196a-48f6-b9ca-d4fd13e22c6d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118962025-04-30T06:26:47.390ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
118972025-04-30T06:26:47.392ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ea1d227-9cc8-4e28-8220-56089d09907a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
118982025-04-30T06:26:47.392ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
118992025-04-30T06:26:47.397ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4126f00-8625-41ba-995c-7a816f99c935", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119002025-04-30T06:26:47.397ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
119012025-04-30T06:26:47.399ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fd743e5-3918-4340-a6e9-3c87c3cf1a1a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119022025-04-30T06:26:47.399ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
119032025-04-30T06:26:47.400ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "728308db-9e03-446f-a45f-eed923e41cbc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119042025-04-30T06:26:47.403ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
119052025-04-30T06:26:47.403ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64f01ca6-3017-4534-8989-cd7a82bccbd0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119062025-04-30T06:26:47.404ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
119072025-04-30T06:26:47.404ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "426c9f12-578b-4a8f-ba2a-4c90e1e53ea1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119082025-04-30T06:26:47.404ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
119092025-04-30T06:26:47.404ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1d217ec-af8e-4118-8cf4-8cb61add9b5a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119102025-04-30T06:26:47.404ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
119112025-04-30T06:26:47.409ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a001bfb0-84d8-477d-855b-934362442ace", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119122025-04-30T06:26:47.409ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
119132025-04-30T06:26:47.412ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6eb0663c-751d-43a8-9edd-94690420e24d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119142025-04-30T06:26:47.412ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
119152025-04-30T06:26:47.416ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
119162025-04-30T06:26:47.418ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e747916-ee37-47b3-a307-035a28ff24a7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119172025-04-30T06:26:47.418ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
119182025-04-30T06:26:47.426ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b4357e5-8239-4cc8-a527-06d0dce80626", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119192025-04-30T06:26:47.427ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
119202025-04-30T06:26:47.429ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1f17af2-9582-4036-879a-e89f8027e068", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119212025-04-30T06:26:47.430ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
119222025-04-30T06:26:47.432ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "428cffca-7b4e-4569-b62c-3d91654973d9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119232025-04-30T06:26:47.432ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
119242025-04-30T06:26:47.434ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "daccc8ef-14b9-4ea6-9b1b-145a0b611912", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119252025-04-30T06:26:47.434ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
119262025-04-30T06:26:47.435ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c0c7793-9dcf-45ca-a88e-8dfd29ad8cdc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119272025-04-30T06:26:47.435ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
119282025-04-30T06:26:47.437ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e5e39c5-62e8-46e1-97e6-dbb29b6e170f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119292025-04-30T06:26:47.437ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
119302025-04-30T06:26:47.439ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddfa77c4-aefa-47f3-99f8-13b69421c579", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119312025-04-30T06:26:47.439ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
119322025-04-30T06:26:47.440ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fd5a7ca-1b2d-4132-8d61-dce8e1074392", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119332025-04-30T06:26:47.440ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119342025-04-30T06:26:47.440ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119352025-04-30T06:26:47.440ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119362025-04-30T06:26:47.440ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119372025-04-30T06:26:47.441ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119382025-04-30T06:26:47.444ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119392025-04-30T06:26:47.452ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
119402025-04-30T06:26:47.452ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119412025-04-30T06:26:47.452ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119422025-04-30T06:26:47.452ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119432025-04-30T06:26:47.452ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119442025-04-30T06:26:47.454ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119452025-04-30T06:26:47.458ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119462025-04-30T06:26:47.465ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
119472025-04-30T06:26:47.465ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119482025-04-30T06:26:47.465ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119492025-04-30T06:26:47.466ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119502025-04-30T06:26:47.466ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119512025-04-30T06:26:47.466ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119522025-04-30T06:26:47.466ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119532025-04-30T06:26:47.466ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
119542025-04-30T06:26:47.466ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119552025-04-30T06:26:47.469ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119562025-04-30T06:26:47.474ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119572025-04-30T06:26:47.474ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119582025-04-30T06:26:47.474ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
119592025-04-30T06:26:47.474ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119602025-04-30T06:26:47.529ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
119612025-04-30T06:26:47.530ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
119622025-04-30T06:26:47.531ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:32792 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
119632025-04-30T06:26:47.531ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:32792 req_id = bccd0d40-9079-4357-9bbc-b8530367ca98 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
119642025-04-30T06:26:47.532ZINFOtest_disk_expunge (dropshot): request completed latency_us = 466 local_addr = [::1]:44559 method = GET remote_addr = [::1]:32792 req_id = bccd0d40-9079-4357-9bbc-b8530367ca98 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
119652025-04-30T06:26:47.532ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119662025-04-30T06:26:47.532ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
119672025-04-30T06:26:47.533ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119682025-04-30T06:26:47.533ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "bccd0d40-9079-4357-9bbc-b8530367ca98", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:46 GMT"} })
119692025-04-30T06:26:47.537ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
119702025-04-30T06:26:47.537ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
119712025-04-30T06:26:47.537ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
119722025-04-30T06:26:47.540ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
119732025-04-30T06:26:47.540ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119742025-04-30T06:26:47.540ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119752025-04-30T06:26:47.540ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119762025-04-30T06:26:47.541ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119772025-04-30T06:26:47.548ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119782025-04-30T06:26:47.554ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
119792025-04-30T06:26:47.555ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119802025-04-30T06:26:47.556ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119812025-04-30T06:26:47.559ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119822025-04-30T06:26:47.617ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
119832025-04-30T06:26:47.618ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
119842025-04-30T06:26:47.621ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
119852025-04-30T06:26:47.621ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 807.816593ms iteration = 42 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
119862025-04-30T06:26:47.621ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 43 reason = Signaled
119872025-04-30T06:26:47.621ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
119882025-04-30T06:26:47.621ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119892025-04-30T06:26:47.621ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119902025-04-30T06:26:47.621ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119912025-04-30T06:26:47.623ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119922025-04-30T06:26:47.646ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119932025-04-30T06:26:47.651ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
119942025-04-30T06:26:47.651ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119952025-04-30T06:26:47.651ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119962025-04-30T06:26:47.652ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119972025-04-30T06:26:47.658ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
119982025-04-30T06:26:47.658ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
119992025-04-30T06:26:47.659ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b9 30 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .0..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63165 req_id = 7eafdaa0-e965-43a9-92e8-a215edbc046c
120002025-04-30T06:26:47.659ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47408,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63165 req_id = 7eafdaa0-e965-43a9-92e8-a215edbc046c
120012025-04-30T06:26:47.659ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
120022025-04-30T06:26:47.659ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120032025-04-30T06:26:47.659ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63165 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7eafdaa0-e965-43a9-92e8-a215edbc046c
120042025-04-30T06:26:47.664ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3272, tv_nsec: 463098489 } })
120052025-04-30T06:26:47.664ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
120062025-04-30T06:26:47.666ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ff a5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49671 req_id = 712c1cdd-9da3-4a52-8854-f50785b2454f
120072025-04-30T06:26:47.666ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65445,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49671 req_id = 712c1cdd-9da3-4a52-8854-f50785b2454f
120082025-04-30T06:26:47.666ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120092025-04-30T06:26:47.667ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49671 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 712c1cdd-9da3-4a52-8854-f50785b2454f
120102025-04-30T06:26:47.776ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
120112025-04-30T06:26:47.777ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
120122025-04-30T06:26:47.777ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120132025-04-30T06:26:47.777ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120142025-04-30T06:26:47.778ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
120152025-04-30T06:26:47.778ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120162025-04-30T06:26:47.780ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: bb b0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61872 req_id = 1fff945d-73ac-4c90-9407-9f135d00b19a
120172025-04-30T06:26:47.780ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48048,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61872 req_id = 1fff945d-73ac-4c90-9407-9f135d00b19a
120182025-04-30T06:26:47.781ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
120192025-04-30T06:26:47.781ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48048, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61872 req_id = 1fff945d-73ac-4c90-9407-9f135d00b19a
120202025-04-30T06:26:47.782ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
120212025-04-30T06:26:47.782ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
120222025-04-30T06:26:47.782ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
120232025-04-30T06:26:47.784ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f76c024-2b16-4d09-9d27-01259e28abe5", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120242025-04-30T06:26:47.787ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
120252025-04-30T06:26:47.788ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120262025-04-30T06:26:47.796ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
120272025-04-30T06:26:47.796ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120282025-04-30T06:26:47.796ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
120292025-04-30T06:26:47.797ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120302025-04-30T06:26:47.800ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1edc497d-af68-4d7b-86ca-c0cb83cf9f55", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120312025-04-30T06:26:47.800ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
120322025-04-30T06:26:47.802ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120332025-04-30T06:26:47.802ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
120342025-04-30T06:26:47.802ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120352025-04-30T06:26:47.807ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01234cc0-d290-4117-8ded-e9dd47f26362", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120362025-04-30T06:26:47.807ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
120372025-04-30T06:26:47.810ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb7f3dd1-1bd2-4bb8-8ee0-f5148ee86c89", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120382025-04-30T06:26:47.811ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
120392025-04-30T06:26:47.815ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d595e21-908e-48fc-adc3-a5041d8e80f2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120402025-04-30T06:26:47.815ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
120412025-04-30T06:26:47.823ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00eebae6-9dde-4e81-a9d5-45df610b3bb6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120422025-04-30T06:26:47.824ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
120432025-04-30T06:26:47.826ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1583211-55c3-4486-b577-b9c7bc9803d0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120442025-04-30T06:26:47.826ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
120452025-04-30T06:26:47.836ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc8cb828-4a3b-4ee0-9c4e-16385879c9fd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120462025-04-30T06:26:47.837ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
120472025-04-30T06:26:47.838ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
120482025-04-30T06:26:47.839ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "054e24a2-a831-4dfb-b728-512cd5e1ac27", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120492025-04-30T06:26:47.840ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
120502025-04-30T06:26:47.841ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e974dc38-03cb-4b6d-8be4-42f4fa3dff3a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120512025-04-30T06:26:47.841ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
120522025-04-30T06:26:47.843ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04ea3683-f4f5-42c3-b36a-8a2b2608fb67", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120532025-04-30T06:26:47.843ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
120542025-04-30T06:26:47.845ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2fbf05a-335d-4fe9-ab1a-f33ea5bcae0a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120552025-04-30T06:26:47.845ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
120562025-04-30T06:26:47.848ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a24ec1b1-ec09-45fd-9145-d80945d957ae", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120572025-04-30T06:26:47.849ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
120582025-04-30T06:26:47.852ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bf76b43-20d8-4c88-a4b2-a7d913205597", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120592025-04-30T06:26:47.852ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
120602025-04-30T06:26:47.854ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "491882e2-204b-4791-956f-1070f1decf06", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120612025-04-30T06:26:47.854ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
120622025-04-30T06:26:47.857ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bca075b-c163-4775-88f5-43201c858a19", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120632025-04-30T06:26:47.857ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
120642025-04-30T06:26:47.859ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62316a16-f14d-46a4-b55f-59267afe206b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120652025-04-30T06:26:47.859ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
120662025-04-30T06:26:47.864ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b69cb305-0675-4889-8b64-04f66cf5f7b9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120672025-04-30T06:26:47.864ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
120682025-04-30T06:26:47.865ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "11e0f4e1-aa16-47a5-97e1-a70dad6c6e19", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120692025-04-30T06:26:47.865ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
120702025-04-30T06:26:47.866ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4801a4c-8c03-4811-b1ca-d71f932b73f0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120712025-04-30T06:26:47.866ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
120722025-04-30T06:26:47.867ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0fa9c72-5528-4ec7-a933-926b2e91440f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120732025-04-30T06:26:47.868ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
120742025-04-30T06:26:47.868ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb97b2e3-af59-4e7a-9fa2-395387d2b6c0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120752025-04-30T06:26:47.869ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
120762025-04-30T06:26:47.870ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7a229a6-34b7-4f2b-9ec6-8deef0692e05", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120772025-04-30T06:26:47.870ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
120782025-04-30T06:26:47.872ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5082c2cf-b540-439c-8ecf-b9b0b8e6da4b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120792025-04-30T06:26:47.872ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
120802025-04-30T06:26:47.874ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c026b03-a024-4db9-957e-28108f879517", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120812025-04-30T06:26:47.874ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
120822025-04-30T06:26:47.875ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25296b7f-541f-4806-aa2e-7d859a4d8776", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120832025-04-30T06:26:47.876ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
120842025-04-30T06:26:47.877ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8af42b42-943d-47f3-aece-8a57099e3db6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120852025-04-30T06:26:47.877ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
120862025-04-30T06:26:47.879ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db148db8-f999-4e63-874a-921c5610c07f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120872025-04-30T06:26:47.879ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
120882025-04-30T06:26:47.881ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4f8cbc8-7133-4359-9828-60856788d4e5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120892025-04-30T06:26:47.882ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
120902025-04-30T06:26:47.884ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be07ec9c-256e-41de-a48a-d56f103d69ae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120912025-04-30T06:26:47.884ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
120922025-04-30T06:26:47.886ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "060916d1-aba1-4e7b-8440-57d0bf1f14e1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120932025-04-30T06:26:47.886ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
120942025-04-30T06:26:47.887ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b92e4736-2c6e-4c06-ad7c-b93e3ef26ade", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120952025-04-30T06:26:47.887ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
120962025-04-30T06:26:47.888ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a9e5c85-91b1-405c-b69d-5ec08ae14c08", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
120972025-04-30T06:26:47.888ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
120982025-04-30T06:26:47.889ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
120992025-04-30T06:26:47.889ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121002025-04-30T06:26:47.889ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121012025-04-30T06:26:47.889ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
121022025-04-30T06:26:47.890ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121032025-04-30T06:26:47.890ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8704c00-0772-4d59-ac49-2dc03310ea73", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121042025-04-30T06:26:47.890ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
121052025-04-30T06:26:47.892ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "37a00f5b-251f-464b-b6e6-f4af2dd5f9a8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121062025-04-30T06:26:47.892ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
121072025-04-30T06:26:47.893ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121082025-04-30T06:26:47.902ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121092025-04-30T06:26:47.902ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121102025-04-30T06:26:47.902ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
121112025-04-30T06:26:47.902ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121122025-04-30T06:26:47.905ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121132025-04-30T06:26:47.906ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
121142025-04-30T06:26:47.906ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121152025-04-30T06:26:47.907ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "044b9635-b290-4bfa-bd40-b1f3009aac85", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121162025-04-30T06:26:47.908ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
121172025-04-30T06:26:47.912ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32a5972a-9537-4ee6-b55a-68cc964be50b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121182025-04-30T06:26:47.912ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
121192025-04-30T06:26:47.919ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6248a81a-7dc6-4380-907a-12875e4f4803", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121202025-04-30T06:26:47.920ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
121212025-04-30T06:26:47.926ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a21fd6e9-a47f-4a51-aaf1-3c7e7805fbc1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121222025-04-30T06:26:47.926ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
121232025-04-30T06:26:47.933ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecacd8a8-3a78-4ae0-a1ca-b48a7e0fbfb4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121242025-04-30T06:26:47.936ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
121252025-04-30T06:26:47.939ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d815e88-d5c7-49e4-bdc2-aa295991cc2e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121262025-04-30T06:26:47.939ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
121272025-04-30T06:26:47.940ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0251682c-4981-4d6f-bff3-600ce3a4e52f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121282025-04-30T06:26:47.940ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
121292025-04-30T06:26:47.942ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c6e9494-a50e-49e7-832e-a813303152d1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121302025-04-30T06:26:47.942ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
121312025-04-30T06:26:47.945ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a573d18-bcb1-4e50-b425-001f3a7ded0d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121322025-04-30T06:26:47.945ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
121332025-04-30T06:26:47.946ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
121342025-04-30T06:26:47.948ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7e98eae-cb17-446e-9d33-ecfe87cc4e60", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121352025-04-30T06:26:47.949ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
121362025-04-30T06:26:47.949ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121372025-04-30T06:26:47.949ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121382025-04-30T06:26:47.949ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121392025-04-30T06:26:47.950ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121402025-04-30T06:26:47.953ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121412025-04-30T06:26:47.962ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
121422025-04-30T06:26:47.962ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
121432025-04-30T06:26:47.962ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121442025-04-30T06:26:47.962ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121452025-04-30T06:26:47.962ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121462025-04-30T06:26:47.963ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121472025-04-30T06:26:47.963ZDEBGtest_disk_expunge (results-sink): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
121482025-04-30T06:26:47.963ZDEBGtest_disk_expunge (results-sink-cluster): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
121492025-04-30T06:26:47.964ZDEBGtest_disk_expunge (clickhouse-client): unrolling 6 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
121502025-04-30T06:26:47.971ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
121512025-04-30T06:26:47.974ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121522025-04-30T06:26:47.981ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
121532025-04-30T06:26:47.981ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121542025-04-30T06:26:47.981ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121552025-04-30T06:26:47.983ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121562025-04-30T06:26:47.985ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 12 table_name = fields_string
121572025-04-30T06:26:47.986ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
121582025-04-30T06:26:48.054ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
121592025-04-30T06:26:48.054ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
121602025-04-30T06:26:48.055ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121612025-04-30T06:26:48.055ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121622025-04-30T06:26:48.055ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121632025-04-30T06:26:48.055ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
121642025-04-30T06:26:48.057ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121652025-04-30T06:26:48.057ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:51172 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
121662025-04-30T06:26:48.058ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121672025-04-30T06:26:48.067ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121682025-04-30T06:26:48.069ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 5 table_name = fields_u16
121692025-04-30T06:26:48.070ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
121702025-04-30T06:26:48.070ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:51172 req_id = 2a7d1504-ef97-4011-8982-5a110cc9c106 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
121712025-04-30T06:26:48.072ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
121722025-04-30T06:26:48.072ZINFOtest_disk_expunge (dropshot): request completed latency_us = 2110 local_addr = [::1]:44559 method = GET remote_addr = [::1]:51172 req_id = 2a7d1504-ef97-4011-8982-5a110cc9c106 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
121732025-04-30T06:26:48.074ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a7d1504-ef97-4011-8982-5a110cc9c106", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
121742025-04-30T06:26:48.078ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
121752025-04-30T06:26:48.078ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
121762025-04-30T06:26:48.078ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
121772025-04-30T06:26:48.081ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
121782025-04-30T06:26:48.081ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121792025-04-30T06:26:48.081ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121802025-04-30T06:26:48.081ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121812025-04-30T06:26:48.082ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121822025-04-30T06:26:48.082ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121832025-04-30T06:26:48.083ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121842025-04-30T06:26:48.083ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121852025-04-30T06:26:48.083ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121862025-04-30T06:26:48.085ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121872025-04-30T06:26:48.091ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121882025-04-30T06:26:48.091ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121892025-04-30T06:26:48.091ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
121902025-04-30T06:26:48.092ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121912025-04-30T06:26:48.093ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 5 table_name = fields_uuid
121922025-04-30T06:26:48.093ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
121932025-04-30T06:26:48.094ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121942025-04-30T06:26:48.098ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
121952025-04-30T06:26:48.098ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
121962025-04-30T06:26:48.098ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
121972025-04-30T06:26:48.099ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
121982025-04-30T06:26:48.104ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121992025-04-30T06:26:48.106ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
122002025-04-30T06:26:48.108ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122012025-04-30T06:26:48.111ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 5 table_name = measurements_histogramu64
122022025-04-30T06:26:48.111ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
122032025-04-30T06:26:48.124ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
122042025-04-30T06:26:48.124ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
122052025-04-30T06:26:48.160ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
122062025-04-30T06:26:48.161ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
122072025-04-30T06:26:48.161ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 542.844742ms iteration = 43 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
122082025-04-30T06:26:48.161ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 44 reason = Signaled
122092025-04-30T06:26:48.161ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122102025-04-30T06:26:48.161ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122112025-04-30T06:26:48.161ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122122025-04-30T06:26:48.164ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122132025-04-30T06:26:48.167ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122142025-04-30T06:26:48.173ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122152025-04-30T06:26:48.189ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122162025-04-30T06:26:48.189ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122172025-04-30T06:26:48.189ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122182025-04-30T06:26:48.189ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122192025-04-30T06:26:48.202ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
122202025-04-30T06:26:48.202ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
122212025-04-30T06:26:48.205ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a7 5c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .\\..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59739 req_id = 14b555bb-18c1-4c64-9d2c-d09806bd0964
122222025-04-30T06:26:48.205ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42844,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59739 req_id = 14b555bb-18c1-4c64-9d2c-d09806bd0964
122232025-04-30T06:26:48.205ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
122242025-04-30T06:26:48.205ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122252025-04-30T06:26:48.205ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59739 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 14b555bb-18c1-4c64-9d2c-d09806bd0964
122262025-04-30T06:26:48.210ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
122272025-04-30T06:26:48.210ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3273, tv_nsec: 9684486 } })
122282025-04-30T06:26:48.210ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
122292025-04-30T06:26:48.212ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a9 23 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .#..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60491 req_id = 7cc68829-3237-40a5-a143-810614e8ab22
122302025-04-30T06:26:48.212ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43299,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60491 req_id = 7cc68829-3237-40a5-a143-810614e8ab22
122312025-04-30T06:26:48.212ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
122322025-04-30T06:26:48.212ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60491 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7cc68829-3237-40a5-a143-810614e8ab22
122332025-04-30T06:26:48.322ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
122342025-04-30T06:26:48.323ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122352025-04-30T06:26:48.323ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122362025-04-30T06:26:48.323ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122372025-04-30T06:26:48.323ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
122382025-04-30T06:26:48.324ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122392025-04-30T06:26:48.324ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2e d9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50775 req_id = 94f534f2-9ee1-43fa-bea8-00be65c02fc8
122402025-04-30T06:26:48.324ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11993,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50775 req_id = 94f534f2-9ee1-43fa-bea8-00be65c02fc8
122412025-04-30T06:26:48.325ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
122422025-04-30T06:26:48.325ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11993, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50775 req_id = 94f534f2-9ee1-43fa-bea8-00be65c02fc8
122432025-04-30T06:26:48.326ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
122442025-04-30T06:26:48.326ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
122452025-04-30T06:26:48.326ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
122462025-04-30T06:26:48.330ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "774ba442-b259-4a15-816c-67bd3c18fc29", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122472025-04-30T06:26:48.330ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
122482025-04-30T06:26:48.332ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122492025-04-30T06:26:48.347ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122502025-04-30T06:26:48.347ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122512025-04-30T06:26:48.347ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
122522025-04-30T06:26:48.347ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122532025-04-30T06:26:48.352ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122542025-04-30T06:26:48.352ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
122552025-04-30T06:26:48.353ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122562025-04-30T06:26:48.353ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a94c525a-dc24-4ce8-ad44-986e50922417", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122572025-04-30T06:26:48.353ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
122582025-04-30T06:26:48.364ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "913d8fe5-bce6-4f6e-9462-2d98ee0ec321", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122592025-04-30T06:26:48.365ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
122602025-04-30T06:26:48.369ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "672942a2-9052-441e-a918-79a1ffbc9737", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122612025-04-30T06:26:48.370ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
122622025-04-30T06:26:48.377ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97d010bc-1b01-4974-b8c4-f10047eb4038", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122632025-04-30T06:26:48.377ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
122642025-04-30T06:26:48.382ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86e81c35-d0d5-4432-98e2-b61a1d0fcea1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122652025-04-30T06:26:48.382ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
122662025-04-30T06:26:48.401ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ca84dd0-88dc-4a91-b5c0-bc5d11bbd0f8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122672025-04-30T06:26:48.404ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
122682025-04-30T06:26:48.406ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
122692025-04-30T06:26:48.410ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c329698-9e9e-4cb5-8fcd-d32310b9c140", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122702025-04-30T06:26:48.410ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
122712025-04-30T06:26:48.412ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1c26b6e-d622-4d66-9d26-8c04a3a8b413", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122722025-04-30T06:26:48.412ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
122732025-04-30T06:26:48.414ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa800cdc-c757-4b80-a80a-ae6e1a58aa26", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122742025-04-30T06:26:48.414ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
122752025-04-30T06:26:48.416ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "616f40d5-a6f2-4423-83ca-68cfb841bbff", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122762025-04-30T06:26:48.416ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
122772025-04-30T06:26:48.418ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55739d34-0f54-48b9-9a75-af61908b4578", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122782025-04-30T06:26:48.418ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
122792025-04-30T06:26:48.421ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00d647ef-0fb6-45d0-abf0-5fbb13cd5a39", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122802025-04-30T06:26:48.421ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
122812025-04-30T06:26:48.426ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e208ccd-f41a-4cc7-b8e9-319075e1e6e1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122822025-04-30T06:26:48.426ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
122832025-04-30T06:26:48.433ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b27a39df-674c-4f10-9219-45e5d94c0adf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122842025-04-30T06:26:48.433ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
122852025-04-30T06:26:48.446ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c0328d5-5bab-4b56-a604-b6de2ea6cc5b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122862025-04-30T06:26:48.446ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
122872025-04-30T06:26:48.455ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d813a7ea-3724-4724-8ac7-c7e075c22783", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
122882025-04-30T06:26:48.456ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
122892025-04-30T06:26:48.456ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122902025-04-30T06:26:48.457ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122912025-04-30T06:26:48.457ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122922025-04-30T06:26:48.457ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
122932025-04-30T06:26:48.458ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122942025-04-30T06:26:48.463ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122952025-04-30T06:26:48.473ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122962025-04-30T06:26:48.473ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122972025-04-30T06:26:48.473ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
122982025-04-30T06:26:48.475ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122992025-04-30T06:26:48.477ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d765bc8c-9934-41ec-b009-1f35a6b79279", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123002025-04-30T06:26:48.477ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
123012025-04-30T06:26:48.478ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123022025-04-30T06:26:48.478ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
123032025-04-30T06:26:48.479ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123042025-04-30T06:26:48.489ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f73daea-e7c0-4169-86d5-9bbe33767d51", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123052025-04-30T06:26:48.490ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
123062025-04-30T06:26:48.504ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e87a8d33-7166-429f-9ab6-12547f66946d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123072025-04-30T06:26:48.504ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
123082025-04-30T06:26:48.508ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e781a2a-94ad-4131-8d7b-49e017a362d6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123092025-04-30T06:26:48.509ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
123102025-04-30T06:26:48.510ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7d3593d-e4fb-4ee8-a4c5-07e8a403575a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123112025-04-30T06:26:48.511ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
123122025-04-30T06:26:48.513ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5ff9ab3-bf69-40e3-8c87-a293f6ef6788", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123132025-04-30T06:26:48.513ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
123142025-04-30T06:26:48.516ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06d98445-b5b8-48ed-b6f2-f4bcba204db1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123152025-04-30T06:26:48.519ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
123162025-04-30T06:26:48.559ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
123172025-04-30T06:26:48.561ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "119ba6b6-e5fb-4eb9-ae08-1b5b095dc01c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123182025-04-30T06:26:48.561ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
123192025-04-30T06:26:48.570ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "736fd21b-2cef-462a-a841-2a2a3419f945", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123202025-04-30T06:26:48.570ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
123212025-04-30T06:26:48.573ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecda4aee-f1d3-4487-8bc2-cba613ce7b0e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123222025-04-30T06:26:48.574ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
123232025-04-30T06:26:48.577ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "368d53bd-e829-45a2-be8c-b835941574aa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123242025-04-30T06:26:48.578ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
123252025-04-30T06:26:48.585ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05127441-004d-4b7a-9601-df265d277be9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123262025-04-30T06:26:48.585ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
123272025-04-30T06:26:48.591ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d849140-1a13-472a-8135-bcb1d5957455", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123282025-04-30T06:26:48.591ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
123292025-04-30T06:26:48.593ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97ffa028-0c53-4cb3-a75c-9f6480335b28", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123302025-04-30T06:26:48.594ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
123312025-04-30T06:26:48.597ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80220d58-a8bb-42aa-b661-324704c3adf3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123322025-04-30T06:26:48.598ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
123332025-04-30T06:26:48.600ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5afd4042-c0fa-4ef4-ab11-c5db0aa6c62d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123342025-04-30T06:26:48.600ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
123352025-04-30T06:26:48.601ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "462edbf0-7e27-4a19-acd0-a7a4f0be4c97", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123362025-04-30T06:26:48.602ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
123372025-04-30T06:26:48.604ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98db3267-f3cb-4d59-95c8-88ebab00d2f7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123382025-04-30T06:26:48.605ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
123392025-04-30T06:26:48.609ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bbf238f-bc07-4600-ae00-19db91586782", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123402025-04-30T06:26:48.610ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
123412025-04-30T06:26:48.610ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
123422025-04-30T06:26:48.610ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123432025-04-30T06:26:48.610ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123442025-04-30T06:26:48.610ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
123452025-04-30T06:26:48.611ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123462025-04-30T06:26:48.615ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123472025-04-30T06:26:48.629ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123482025-04-30T06:26:48.630ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123492025-04-30T06:26:48.630ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
123502025-04-30T06:26:48.630ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123512025-04-30T06:26:48.644ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123522025-04-30T06:26:48.644ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
123532025-04-30T06:26:48.644ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123542025-04-30T06:26:48.645ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cdb7224-0f0e-4afb-a50a-afb2eb84ea91", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123552025-04-30T06:26:48.647ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
123562025-04-30T06:26:48.659ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c26fc42-1a6a-4309-8f2e-f86d87545984", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123572025-04-30T06:26:48.659ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
123582025-04-30T06:26:48.668ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e9a6ff2-9ce5-486f-81df-05e27404dfe6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123592025-04-30T06:26:48.668ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
123602025-04-30T06:26:48.679ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b23b60ee-d0bb-4a42-a962-aed258ddc6b5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123612025-04-30T06:26:48.679ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
123622025-04-30T06:26:48.705ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6272d43-0bb6-460a-afd0-950bd6ad002e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123632025-04-30T06:26:48.706ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
123642025-04-30T06:26:48.711ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aec30c8a-dc55-4a03-9765-12ef4925d0c8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123652025-04-30T06:26:48.712ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
123662025-04-30T06:26:48.712ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
123672025-04-30T06:26:48.714ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "68ad7ad8-5249-483a-85e1-4795c8f2869b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123682025-04-30T06:26:48.715ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
123692025-04-30T06:26:48.716ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "148613a2-2c69-4436-a604-06027269d180", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123702025-04-30T06:26:48.716ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
123712025-04-30T06:26:48.717ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b747382a-ff07-4e8e-bd72-a1495bb27560", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:47 GMT"} })
123722025-04-30T06:26:48.718ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
123732025-04-30T06:26:48.718ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123742025-04-30T06:26:48.718ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123752025-04-30T06:26:48.718ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123762025-04-30T06:26:48.720ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123772025-04-30T06:26:48.723ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123782025-04-30T06:26:48.731ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
123792025-04-30T06:26:48.732ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
123802025-04-30T06:26:48.733ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123812025-04-30T06:26:48.733ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123822025-04-30T06:26:48.733ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123832025-04-30T06:26:48.734ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123842025-04-30T06:26:48.738ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123852025-04-30T06:26:48.751ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
123862025-04-30T06:26:48.752ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123872025-04-30T06:26:48.752ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123882025-04-30T06:26:48.752ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123892025-04-30T06:26:48.836ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
123902025-04-30T06:26:48.836ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
123912025-04-30T06:26:48.839ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
123922025-04-30T06:26:48.839ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123932025-04-30T06:26:48.839ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123942025-04-30T06:26:48.839ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
123952025-04-30T06:26:48.840ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123962025-04-30T06:26:48.840ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:47213 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
123972025-04-30T06:26:48.841ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:47213 req_id = 28de91b7-99c3-4dd3-9057-a620d0f1062a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
123982025-04-30T06:26:48.845ZINFOtest_disk_expunge (dropshot): request completed latency_us = 3820 local_addr = [::1]:44559 method = GET remote_addr = [::1]:47213 req_id = 28de91b7-99c3-4dd3-9057-a620d0f1062a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
123992025-04-30T06:26:48.845ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "28de91b7-99c3-4dd3-9057-a620d0f1062a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
124002025-04-30T06:26:48.849ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
124012025-04-30T06:26:48.850ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
124022025-04-30T06:26:48.850ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
124032025-04-30T06:26:48.853ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
124042025-04-30T06:26:48.853ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124052025-04-30T06:26:48.853ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124062025-04-30T06:26:48.853ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124072025-04-30T06:26:48.855ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124082025-04-30T06:26:48.857ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124092025-04-30T06:26:48.867ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
124102025-04-30T06:26:48.867ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124112025-04-30T06:26:48.867ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
124122025-04-30T06:26:48.868ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124132025-04-30T06:26:48.870ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124142025-04-30T06:26:48.881ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
124152025-04-30T06:26:48.882ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124162025-04-30T06:26:48.882ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124172025-04-30T06:26:48.883ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124182025-04-30T06:26:48.884ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
124192025-04-30T06:26:48.885ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
124202025-04-30T06:26:48.885ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 0bb2bdac-58d4-463f-a028-080dca66cb0b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
124212025-04-30T06:26:48.886ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 0bb2bdac-58d4-463f-a028-080dca66cb0b resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
124222025-04-30T06:26:48.886ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 0bb2bdac-58d4-463f-a028-080dca66cb0b roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
124232025-04-30T06:26:48.887ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 0bb2bdac-58d4-463f-a028-080dca66cb0b resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
124242025-04-30T06:26:48.887ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124252025-04-30T06:26:48.887ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
124262025-04-30T06:26:48.888ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124272025-04-30T06:26:48.900ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 13627 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 0bb2bdac-58d4-463f-a028-080dca66cb0b response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
124282025-04-30T06:26:48.906ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bb2bdac-58d4-463f-a028-080dca66cb0b", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
124292025-04-30T06:26:48.906ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
124302025-04-30T06:26:48.908ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 025c5031-2246-43bd-8f27-6558af8445a1 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
124312025-04-30T06:26:48.909ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 025c5031-2246-43bd-8f27-6558af8445a1 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
124322025-04-30T06:26:48.909ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 025c5031-2246-43bd-8f27-6558af8445a1 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
124332025-04-30T06:26:48.909ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 025c5031-2246-43bd-8f27-6558af8445a1 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
124342025-04-30T06:26:48.914ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 5631 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 025c5031-2246-43bd-8f27-6558af8445a1 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
124352025-04-30T06:26:48.915ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "025c5031-2246-43bd-8f27-6558af8445a1", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
124362025-04-30T06:26:48.915ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
124372025-04-30T06:26:48.915ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124382025-04-30T06:26:48.915ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
124392025-04-30T06:26:49.017ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124402025-04-30T06:26:49.017ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124412025-04-30T06:26:49.019ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
124422025-04-30T06:26:49.032ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
124432025-04-30T06:26:49.035ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
124442025-04-30T06:26:49.036ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
124452025-04-30T06:26:49.036ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 874.651947ms iteration = 44 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
124462025-04-30T06:26:49.036ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 45 reason = Signaled
124472025-04-30T06:26:49.036ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
124482025-04-30T06:26:49.036ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124492025-04-30T06:26:49.036ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124502025-04-30T06:26:49.036ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124512025-04-30T06:26:49.037ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124522025-04-30T06:26:49.046ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
124532025-04-30T06:26:49.046ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
124542025-04-30T06:26:49.047ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124552025-04-30T06:26:49.047ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124562025-04-30T06:26:49.047ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124572025-04-30T06:26:49.052ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
124582025-04-30T06:26:49.052ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
124592025-04-30T06:26:49.052ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
124602025-04-30T06:26:49.052ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
124612025-04-30T06:26:49.063ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
124622025-04-30T06:26:49.074ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
124632025-04-30T06:26:49.074ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124642025-04-30T06:26:49.075ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124652025-04-30T06:26:49.075ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124662025-04-30T06:26:49.075ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124672025-04-30T06:26:49.075ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124682025-04-30T06:26:49.076ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
124692025-04-30T06:26:49.076ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
124702025-04-30T06:26:49.082ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: db ba 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:54352 req_id = 66ae2816-b722-4970-8f5a-870ff8646860
124712025-04-30T06:26:49.083ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56250,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54352 req_id = 66ae2816-b722-4970-8f5a-870ff8646860
124722025-04-30T06:26:49.083ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
124732025-04-30T06:26:49.084ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124742025-04-30T06:26:49.084ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54352 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 66ae2816-b722-4970-8f5a-870ff8646860
124752025-04-30T06:26:49.085ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
124762025-04-30T06:26:49.096ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3273, tv_nsec: 894942845 } })
124772025-04-30T06:26:49.096ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
124782025-04-30T06:26:49.096ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f8 8a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54365 req_id = b2afb1e8-473f-44a9-b30f-d06e9fb22c63
124792025-04-30T06:26:49.097ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63626,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54365 req_id = b2afb1e8-473f-44a9-b30f-d06e9fb22c63
124802025-04-30T06:26:49.097ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
124812025-04-30T06:26:49.097ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124822025-04-30T06:26:49.097ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54365 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b2afb1e8-473f-44a9-b30f-d06e9fb22c63
124832025-04-30T06:26:49.099ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3273, tv_nsec: 898890310 } })
124842025-04-30T06:26:49.100ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
124852025-04-30T06:26:49.108ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 75 fb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e u...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54882 req_id = 3d18a406-024f-4c9e-97c0-7cffd32808e6
124862025-04-30T06:26:49.108ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30203,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54882 req_id = 3d18a406-024f-4c9e-97c0-7cffd32808e6
124872025-04-30T06:26:49.108ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124882025-04-30T06:26:49.108ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54882 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3d18a406-024f-4c9e-97c0-7cffd32808e6
124892025-04-30T06:26:49.109ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
124902025-04-30T06:26:49.156ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
124912025-04-30T06:26:49.156ZDEBGtest_disk_expunge (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
124922025-04-30T06:26:49.156ZDEBGtest_disk_expunge (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
124932025-04-30T06:26:49.156ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
124942025-04-30T06:26:49.157ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
124952025-04-30T06:26:49.159ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f5 d1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62185 req_id = 158b2d72-8083-430e-a5fd-aa6a5bafe94c
124962025-04-30T06:26:49.160ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62929,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62185 req_id = 158b2d72-8083-430e-a5fd-aa6a5bafe94c
124972025-04-30T06:26:49.160ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124982025-04-30T06:26:49.160ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62185 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 158b2d72-8083-430e-a5fd-aa6a5bafe94c
124992025-04-30T06:26:49.161ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125002025-04-30T06:26:49.161ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125012025-04-30T06:26:49.161ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125022025-04-30T06:26:49.161ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
125032025-04-30T06:26:49.162ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125042025-04-30T06:26:49.240ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
125052025-04-30T06:26:49.241ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
125062025-04-30T06:26:49.241ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
125072025-04-30T06:26:49.244ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 97 56 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .V..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62426 req_id = 41de5110-be29-4e47-87e3-a6653e06bc45
125082025-04-30T06:26:49.244ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38742,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62426 req_id = 41de5110-be29-4e47-87e3-a6653e06bc45
125092025-04-30T06:26:49.245ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
125102025-04-30T06:26:49.245ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38742, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62426 req_id = 41de5110-be29-4e47-87e3-a6653e06bc45
125112025-04-30T06:26:49.245ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125122025-04-30T06:26:49.254ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125132025-04-30T06:26:49.254ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125142025-04-30T06:26:49.254ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
125152025-04-30T06:26:49.255ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125162025-04-30T06:26:49.255ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
125172025-04-30T06:26:49.255ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
125182025-04-30T06:26:49.255ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
125192025-04-30T06:26:49.260ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: b8 c5 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:54012 req_id = 18aea9d7-1746-4ffc-835e-775ccf95623a
125202025-04-30T06:26:49.261ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47301,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54012 req_id = 18aea9d7-1746-4ffc-835e-775ccf95623a
125212025-04-30T06:26:49.261ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
125222025-04-30T06:26:49.262ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125232025-04-30T06:26:49.262ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54012 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 18aea9d7-1746-4ffc-835e-775ccf95623a
125242025-04-30T06:26:49.264ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3274, tv_nsec: 63029297 } })
125252025-04-30T06:26:49.264ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
125262025-04-30T06:26:49.270ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125272025-04-30T06:26:49.270ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
125282025-04-30T06:26:49.271ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125292025-04-30T06:26:49.271ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e1 1b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59143 req_id = 6e6dc92e-9939-45d6-b88b-2306596b27b9
125302025-04-30T06:26:49.271ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57627,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59143 req_id = 6e6dc92e-9939-45d6-b88b-2306596b27b9
125312025-04-30T06:26:49.271ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125322025-04-30T06:26:49.271ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59143 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6e6dc92e-9939-45d6-b88b-2306596b27b9
125332025-04-30T06:26:49.272ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
125342025-04-30T06:26:49.338ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
125352025-04-30T06:26:49.338ZDEBGtest_disk_expunge (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
125362025-04-30T06:26:49.339ZDEBGtest_disk_expunge (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
125372025-04-30T06:26:49.339ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
125382025-04-30T06:26:49.341ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
125392025-04-30T06:26:49.352ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
125402025-04-30T06:26:49.364ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ac6d2a8-6460-4228-be47-7bacb195f696", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125412025-04-30T06:26:49.365ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
125422025-04-30T06:26:49.367ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: f9 95 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:51047 req_id = ec7d7fdb-e0db-4a75-ac06-b596ff392a8e
125432025-04-30T06:26:49.367ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63893,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51047 req_id = ec7d7fdb-e0db-4a75-ac06-b596ff392a8e
125442025-04-30T06:26:49.367ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
125452025-04-30T06:26:49.367ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125462025-04-30T06:26:49.367ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51047 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ec7d7fdb-e0db-4a75-ac06-b596ff392a8e
125472025-04-30T06:26:49.368ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3274, tv_nsec: 167335029 } })
125482025-04-30T06:26:49.368ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
125492025-04-30T06:26:49.370ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ff ff 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49718 req_id = 8e95fffc-3da0-459b-a0f4-70f52eb7a4e8
125502025-04-30T06:26:49.370ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65535,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49718 req_id = 8e95fffc-3da0-459b-a0f4-70f52eb7a4e8
125512025-04-30T06:26:49.370ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125522025-04-30T06:26:49.370ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49718 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8e95fffc-3da0-459b-a0f4-70f52eb7a4e8
125532025-04-30T06:26:49.372ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
125542025-04-30T06:26:49.499ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
125552025-04-30T06:26:49.499ZDEBGtest_disk_expunge (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
125562025-04-30T06:26:49.505ZDEBGtest_disk_expunge (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
125572025-04-30T06:26:49.505ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
125582025-04-30T06:26:49.509ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
125592025-04-30T06:26:49.510ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fd4cad0-bbd6-4ef5-83df-ce70326213b4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125602025-04-30T06:26:49.510ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
125612025-04-30T06:26:49.524ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d97036c1-e349-4332-b77c-1c303ee89f6b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125622025-04-30T06:26:49.524ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
125632025-04-30T06:26:49.531ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e1ab652-fa64-4886-a29d-697a678fda6d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125642025-04-30T06:26:49.531ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
125652025-04-30T06:26:49.537ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83592e01-b9a6-4c08-97c5-e92c8ac010d1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125662025-04-30T06:26:49.537ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
125672025-04-30T06:26:49.542ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09754afa-f6d4-4bae-bc84-f967b653f89e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125682025-04-30T06:26:49.542ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
125692025-04-30T06:26:49.548ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
125702025-04-30T06:26:49.550ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0d05990-3bb1-4b27-840d-44ccd55b6397", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125712025-04-30T06:26:49.567ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
125722025-04-30T06:26:49.583ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d080cf6-612f-43bb-9192-3e21cdadb057", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125732025-04-30T06:26:49.584ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
125742025-04-30T06:26:49.586ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a49289f8-1cdc-4d32-8069-2f05c597afff", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125752025-04-30T06:26:49.587ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
125762025-04-30T06:26:49.591ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0811d2d-093b-40c9-a238-25a5f91dd8bd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125772025-04-30T06:26:49.591ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
125782025-04-30T06:26:49.595ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f577efd1-f06e-4a98-b670-3a0be4b2ee0a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125792025-04-30T06:26:49.595ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
125802025-04-30T06:26:49.597ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba1a07c7-ac68-4830-9e32-324bee5a5f4c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125812025-04-30T06:26:49.597ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
125822025-04-30T06:26:49.600ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125832025-04-30T06:26:49.600ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125842025-04-30T06:26:49.600ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125852025-04-30T06:26:49.600ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
125862025-04-30T06:26:49.601ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125872025-04-30T06:26:49.601ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "311d6b6f-815f-4bd5-bb50-0a1459796f38", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125882025-04-30T06:26:49.603ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
125892025-04-30T06:26:49.607ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125902025-04-30T06:26:49.622ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125912025-04-30T06:26:49.622ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125922025-04-30T06:26:49.622ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
125932025-04-30T06:26:49.622ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125942025-04-30T06:26:49.627ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e112f45-19a1-4d48-93a2-9b4c837284de", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
125952025-04-30T06:26:49.627ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
125962025-04-30T06:26:49.628ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125972025-04-30T06:26:49.628ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
125982025-04-30T06:26:49.629ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125992025-04-30T06:26:49.654ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c95cf2dc-e38a-48eb-9ef7-668efc1090f6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126002025-04-30T06:26:49.654ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
126012025-04-30T06:26:49.657ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "af601e2d-1e45-4bad-8ed7-fcccd438d5e7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126022025-04-30T06:26:49.657ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
126032025-04-30T06:26:49.665ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc84dc8f-14ab-4e10-b43b-cdb807fb26d4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126042025-04-30T06:26:49.665ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
126052025-04-30T06:26:49.668ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6579f367-33bc-4ff7-8841-8f9da5830dee", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126062025-04-30T06:26:49.669ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
126072025-04-30T06:26:49.674ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1773500d-b1ad-4409-9520-b712fde05c78", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126082025-04-30T06:26:49.674ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
126092025-04-30T06:26:49.676ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a07f8f6f-bd68-4869-b984-79621591f9dd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126102025-04-30T06:26:49.676ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
126112025-04-30T06:26:49.679ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae5d931e-9826-439e-b137-1003a4ee9109", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126122025-04-30T06:26:49.679ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
126132025-04-30T06:26:49.681ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
126142025-04-30T06:26:49.682ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33ed0911-1f37-4fe8-a2aa-136fe3ae7246", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126152025-04-30T06:26:49.682ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
126162025-04-30T06:26:49.685ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e4e5b7b-f3cc-46cc-9381-1847e14c5980", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126172025-04-30T06:26:49.685ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
126182025-04-30T06:26:49.686ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d01f33cd-8431-4f65-81b4-bdaa016e690c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126192025-04-30T06:26:49.687ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
126202025-04-30T06:26:49.689ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0f69a7a-a3a0-4fe3-997c-bab572ae613f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126212025-04-30T06:26:49.689ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
126222025-04-30T06:26:49.691ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b39537fe-1539-4abc-b634-b84f6c3996f9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126232025-04-30T06:26:49.691ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
126242025-04-30T06:26:49.694ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "914ebdc9-aff2-4417-b98e-12470ddf2f13", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126252025-04-30T06:26:49.694ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
126262025-04-30T06:26:49.699ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b717d64-5fc5-4d45-9945-1c3d36e1bd10", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126272025-04-30T06:26:49.699ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
126282025-04-30T06:26:49.702ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0419e4b3-bb78-4aca-884d-8308b2dc34a7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126292025-04-30T06:26:49.702ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
126302025-04-30T06:26:49.706ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f63ebaa4-d830-4fec-bc27-d89d962c2147", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126312025-04-30T06:26:49.707ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
126322025-04-30T06:26:49.708ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c25542b-3574-4e4c-b7e4-1ba189c886c1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126332025-04-30T06:26:49.708ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
126342025-04-30T06:26:49.709ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0c914b1-ebe9-4246-a918-075aa9f7ae8e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126352025-04-30T06:26:49.710ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
126362025-04-30T06:26:49.711ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1890df3-3f45-45f4-bd5c-e297431a078a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126372025-04-30T06:26:49.711ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
126382025-04-30T06:26:49.713ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e72b3ff4-f6c1-457d-b5f4-1c1f074fd4c0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126392025-04-30T06:26:49.713ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
126402025-04-30T06:26:49.717ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a96869c-cfc0-4da0-82d3-32ef258fd2e3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126412025-04-30T06:26:49.717ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
126422025-04-30T06:26:49.719ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b7c0e7e-ce43-4dba-ae2e-9c829029fa09", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126432025-04-30T06:26:49.720ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
126442025-04-30T06:26:49.722ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b996908e-cef9-4343-903b-ba95b5e3368d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126452025-04-30T06:26:49.724ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
126462025-04-30T06:26:49.726ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5491500e-0355-4762-903e-82b5c70db289", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126472025-04-30T06:26:49.726ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
126482025-04-30T06:26:49.729ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c712498-2e7c-456c-8f1f-173d21b78d3e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126492025-04-30T06:26:49.730ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
126502025-04-30T06:26:49.732ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdd2a2b1-79fc-425a-b1f6-0e1aa36074ae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126512025-04-30T06:26:49.732ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
126522025-04-30T06:26:49.733ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126532025-04-30T06:26:49.733ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126542025-04-30T06:26:49.733ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126552025-04-30T06:26:49.733ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
126562025-04-30T06:26:49.734ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126572025-04-30T06:26:49.742ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "200d4ca7-b6f9-427c-a5c7-89a38d710e59", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126582025-04-30T06:26:49.742ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
126592025-04-30T06:26:49.744ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f22ee11-3cf3-47a7-b9b1-80eef9cb7fcd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126602025-04-30T06:26:49.745ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
126612025-04-30T06:26:49.745ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126622025-04-30T06:26:49.766ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126632025-04-30T06:26:49.766ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126642025-04-30T06:26:49.766ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
126652025-04-30T06:26:49.766ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126662025-04-30T06:26:49.769ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8abd3a2a-3945-4aee-b0f3-3139231c99ac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126672025-04-30T06:26:49.771ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
126682025-04-30T06:26:49.773ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c485b15e-e288-4187-87d9-b82495dffe2b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126692025-04-30T06:26:49.773ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
126702025-04-30T06:26:49.773ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126712025-04-30T06:26:49.773ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
126722025-04-30T06:26:49.774ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126732025-04-30T06:26:49.777ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c3f95af-efcf-46de-a0f1-50ec0b1b26f8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:48 GMT"} })
126742025-04-30T06:26:49.777ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
126752025-04-30T06:26:49.777ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126762025-04-30T06:26:49.777ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126772025-04-30T06:26:49.777ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126782025-04-30T06:26:49.779ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126792025-04-30T06:26:49.783ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126802025-04-30T06:26:49.796ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
126812025-04-30T06:26:49.796ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
126822025-04-30T06:26:49.796ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126832025-04-30T06:26:49.796ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126842025-04-30T06:26:49.796ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126852025-04-30T06:26:49.797ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126862025-04-30T06:26:49.806ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126872025-04-30T06:26:49.820ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
126882025-04-30T06:26:49.820ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126892025-04-30T06:26:49.820ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126902025-04-30T06:26:49.821ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126912025-04-30T06:26:49.922ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
126922025-04-30T06:26:49.922ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
126932025-04-30T06:26:49.922ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:61946 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
126942025-04-30T06:26:49.923ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:61946 req_id = 1e495352-1569-4948-a475-70f95c4c4ed2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
126952025-04-30T06:26:49.925ZINFOtest_disk_expunge (dropshot): request completed latency_us = 2380 local_addr = [::1]:44559 method = GET remote_addr = [::1]:61946 req_id = 1e495352-1569-4948-a475-70f95c4c4ed2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
126962025-04-30T06:26:49.926ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e495352-1569-4948-a475-70f95c4c4ed2", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
126972025-04-30T06:26:49.931ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
126982025-04-30T06:26:49.931ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
126992025-04-30T06:26:49.931ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
127002025-04-30T06:26:49.933ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
127012025-04-30T06:26:49.933ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127022025-04-30T06:26:49.933ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127032025-04-30T06:26:49.933ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127042025-04-30T06:26:49.934ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127052025-04-30T06:26:49.942ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127062025-04-30T06:26:49.948ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
127072025-04-30T06:26:49.948ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127082025-04-30T06:26:49.948ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127092025-04-30T06:26:49.949ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127102025-04-30T06:26:49.956ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
127112025-04-30T06:26:49.956ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
127122025-04-30T06:26:49.956ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
127132025-04-30T06:26:49.957ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
127142025-04-30T06:26:49.972ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
127152025-04-30T06:26:49.973ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
127162025-04-30T06:26:49.973ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
127172025-04-30T06:26:49.978ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
127182025-04-30T06:26:49.978ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
127192025-04-30T06:26:50.018ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
127202025-04-30T06:26:50.018ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
127212025-04-30T06:26:50.019ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 982.443865ms iteration = 45 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
127222025-04-30T06:26:50.019ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 46 reason = Signaled
127232025-04-30T06:26:50.019ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
127242025-04-30T06:26:50.019ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127252025-04-30T06:26:50.019ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127262025-04-30T06:26:50.019ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127272025-04-30T06:26:50.019ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127282025-04-30T06:26:50.026ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127292025-04-30T06:26:50.026ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127302025-04-30T06:26:50.026ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127312025-04-30T06:26:50.026ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
127322025-04-30T06:26:50.027ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127332025-04-30T06:26:50.027ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127342025-04-30T06:26:50.035ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
127352025-04-30T06:26:50.035ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
127362025-04-30T06:26:50.035ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
127372025-04-30T06:26:50.036ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
127382025-04-30T06:26:50.041ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127392025-04-30T06:26:50.052ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127402025-04-30T06:26:50.052ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127412025-04-30T06:26:50.052ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
127422025-04-30T06:26:50.054ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127432025-04-30T06:26:50.056ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
127442025-04-30T06:26:50.056ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
127452025-04-30T06:26:50.066ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127462025-04-30T06:26:50.077ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127472025-04-30T06:26:50.078ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
127482025-04-30T06:26:50.081ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127492025-04-30T06:26:50.081ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
127502025-04-30T06:26:50.083ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127512025-04-30T06:26:50.086ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 81 45 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .E..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52772 req_id = 79147842-c7ea-4efb-ba7a-4d30da7336d6
127522025-04-30T06:26:50.086ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33093,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52772 req_id = 79147842-c7ea-4efb-ba7a-4d30da7336d6
127532025-04-30T06:26:50.086ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
127542025-04-30T06:26:50.087ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
127552025-04-30T06:26:50.087ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52772 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 79147842-c7ea-4efb-ba7a-4d30da7336d6
127562025-04-30T06:26:50.091ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3274, tv_nsec: 889746366 } })
127572025-04-30T06:26:50.091ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
127582025-04-30T06:26:50.094ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127592025-04-30T06:26:50.096ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127602025-04-30T06:26:50.096ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127612025-04-30T06:26:50.096ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
127622025-04-30T06:26:50.100ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a4 8e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62781 req_id = 9e0cc329-48b8-4058-9616-2698abedce33
127632025-04-30T06:26:50.101ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42126,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62781 req_id = 9e0cc329-48b8-4058-9616-2698abedce33
127642025-04-30T06:26:50.101ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
127652025-04-30T06:26:50.101ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62781 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9e0cc329-48b8-4058-9616-2698abedce33
127662025-04-30T06:26:50.182ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
127672025-04-30T06:26:50.185ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c3 fa 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58107 req_id = 358583f2-62d8-4982-8549-0533df82df16
127682025-04-30T06:26:50.186ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50170,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58107 req_id = 358583f2-62d8-4982-8549-0533df82df16
127692025-04-30T06:26:50.186ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
127702025-04-30T06:26:50.186ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50170, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58107 req_id = 358583f2-62d8-4982-8549-0533df82df16
127712025-04-30T06:26:50.186ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
127722025-04-30T06:26:50.186ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
127732025-04-30T06:26:50.186ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
127742025-04-30T06:26:50.212ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "439c37d1-64e4-413a-921f-0213377e3f92", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127752025-04-30T06:26:50.214ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
127762025-04-30T06:26:50.215ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1caacb52-eb3f-4323-839b-208865329e0e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127772025-04-30T06:26:50.215ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
127782025-04-30T06:26:50.217ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0a2a27f-33eb-4730-b81e-75f279438748", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127792025-04-30T06:26:50.217ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
127802025-04-30T06:26:50.219ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f197acb-1715-4945-b685-c4908f60aa2c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127812025-04-30T06:26:50.219ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
127822025-04-30T06:26:50.221ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "947ab94c-7ecd-4c80-970e-cd2efa9611cf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127832025-04-30T06:26:50.221ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
127842025-04-30T06:26:50.226ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5055a67f-82dd-48f6-b092-6dd48c376b82", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127852025-04-30T06:26:50.226ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
127862025-04-30T06:26:50.231ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff46bcae-c7c3-4103-8adb-5cc966649434", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127872025-04-30T06:26:50.231ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
127882025-04-30T06:26:50.232ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86f87231-df44-4c21-abc8-18a1f1d448d2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127892025-04-30T06:26:50.232ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
127902025-04-30T06:26:50.233ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff55abd2-3498-4b0d-b669-5cfd5d3f09b9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127912025-04-30T06:26:50.234ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
127922025-04-30T06:26:50.236ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0442ec43-35f9-4288-b755-c698e7b9f67e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127932025-04-30T06:26:50.236ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
127942025-04-30T06:26:50.237ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ad3854b-243e-4bf2-a7ba-cfc6cb6ead92", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127952025-04-30T06:26:50.237ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
127962025-04-30T06:26:50.238ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c6fe681-c3c1-4354-a859-8c9a90790d75", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127972025-04-30T06:26:50.238ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
127982025-04-30T06:26:50.240ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a38974ab-eb48-4092-967d-280a992bde1f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
127992025-04-30T06:26:50.240ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
128002025-04-30T06:26:50.242ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8191b75-2622-412d-b329-5555b9035ba9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128012025-04-30T06:26:50.242ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
128022025-04-30T06:26:50.244ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "143bd2a7-c8d7-40fc-90eb-779ca1af81f7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128032025-04-30T06:26:50.244ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
128042025-04-30T06:26:50.246ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad9f3f9c-64d4-4ada-8b21-25e050274234", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128052025-04-30T06:26:50.246ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
128062025-04-30T06:26:50.253ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "991de937-794c-45cd-b175-0551114b4f13", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128072025-04-30T06:26:50.253ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
128082025-04-30T06:26:50.263ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae163ff7-71f7-4301-8f4c-9f68184b7558", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128092025-04-30T06:26:50.263ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
128102025-04-30T06:26:50.266ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "506d75f5-2ead-4067-be34-6b55d0cd0693", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128112025-04-30T06:26:50.266ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
128122025-04-30T06:26:50.267ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76e0ebe5-3f41-473d-8d85-3ad9d0646d3a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128132025-04-30T06:26:50.268ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
128142025-04-30T06:26:50.269ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b8d54e5-7d39-4378-a483-68c1200024a8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128152025-04-30T06:26:50.269ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
128162025-04-30T06:26:50.270ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbfae131-318a-4a23-b52d-56c2e82d71fd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128172025-04-30T06:26:50.271ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
128182025-04-30T06:26:50.273ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c80a8fb9-c5f4-46f8-9e21-07a979cb949c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128192025-04-30T06:26:50.274ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
128202025-04-30T06:26:50.276ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a14a13e-d757-4bd8-9b4c-cfa250ba2242", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128212025-04-30T06:26:50.276ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
128222025-04-30T06:26:50.279ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01920fb0-e209-4d7b-b945-d3b151f9143d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128232025-04-30T06:26:50.279ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
128242025-04-30T06:26:50.283ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a958d596-0589-4375-9bbd-a56ca4a69b3f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128252025-04-30T06:26:50.283ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
128262025-04-30T06:26:50.286ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "08d8e0e7-5efe-4115-bd44-33f2595eb97a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128272025-04-30T06:26:50.287ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
128282025-04-30T06:26:50.289ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec4b272f-f84f-4413-be38-881186f23bbd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128292025-04-30T06:26:50.290ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
128302025-04-30T06:26:50.294ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "987e9d10-225f-485d-b320-7e252d417297", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128312025-04-30T06:26:50.294ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
128322025-04-30T06:26:50.297ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b90db1c9-45dd-41d0-8d19-8a622604094d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128332025-04-30T06:26:50.298ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
128342025-04-30T06:26:50.299ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a76d6359-4d53-408e-8360-d9839fb1ca2c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128352025-04-30T06:26:50.299ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
128362025-04-30T06:26:50.301ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cbe9009-5a94-4df9-a1ac-f62dbee73d9b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128372025-04-30T06:26:50.301ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
128382025-04-30T06:26:50.302ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c32781e-d2f8-4175-8efa-d46970c801e3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128392025-04-30T06:26:50.302ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
128402025-04-30T06:26:50.303ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "efc28352-bc96-46f4-a201-c61d26719e72", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128412025-04-30T06:26:50.304ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
128422025-04-30T06:26:50.306ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7fc771c-06c3-4273-aeff-140bdab3628a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128432025-04-30T06:26:50.306ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
128442025-04-30T06:26:50.311ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77c66f8e-998c-4d0d-93fd-9a3408af9ead", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128452025-04-30T06:26:50.311ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
128462025-04-30T06:26:50.311ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
128472025-04-30T06:26:50.314ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08a3fafc-ef53-4e79-bd6d-716d462dbdeb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128482025-04-30T06:26:50.314ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
128492025-04-30T06:26:50.317ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "84ae5eca-d8e2-4a11-b525-243532f3b60e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128502025-04-30T06:26:50.317ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
128512025-04-30T06:26:50.322ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9936dd55-e112-4ff2-a40e-47e7deed1627", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128522025-04-30T06:26:50.322ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
128532025-04-30T06:26:50.324ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8256ddd-2c55-4c73-84b0-60c44ae15801", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128542025-04-30T06:26:50.324ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
128552025-04-30T06:26:50.327ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b10e400-b324-418a-8364-c9baec63b555", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128562025-04-30T06:26:50.327ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
128572025-04-30T06:26:50.332ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c35a5200-8228-4caf-be6c-167fdcd51fa5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128582025-04-30T06:26:50.332ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
128592025-04-30T06:26:50.332ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e945542-fdb5-4238-8b72-086335490706", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128602025-04-30T06:26:50.332ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
128612025-04-30T06:26:50.333ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f0994fd-919f-450a-8b04-e04726319f99", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128622025-04-30T06:26:50.334ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
128632025-04-30T06:26:50.336ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39f52c8a-d1a3-4aa5-b962-0c48ceb8a70a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128642025-04-30T06:26:50.336ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
128652025-04-30T06:26:50.336ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128662025-04-30T06:26:50.336ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128672025-04-30T06:26:50.336ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128682025-04-30T06:26:50.347ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128692025-04-30T06:26:50.350ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128702025-04-30T06:26:50.357ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128712025-04-30T06:26:50.358ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
128722025-04-30T06:26:50.358ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128732025-04-30T06:26:50.358ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128742025-04-30T06:26:50.358ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128752025-04-30T06:26:50.359ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128762025-04-30T06:26:50.363ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128772025-04-30T06:26:50.371ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128782025-04-30T06:26:50.371ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128792025-04-30T06:26:50.371ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128802025-04-30T06:26:50.373ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128812025-04-30T06:26:50.373ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128822025-04-30T06:26:50.374ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128832025-04-30T06:26:50.374ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128842025-04-30T06:26:50.374ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
128852025-04-30T06:26:50.375ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128862025-04-30T06:26:50.441ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
128872025-04-30T06:26:50.441ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
128882025-04-30T06:26:50.442ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:65300 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
128892025-04-30T06:26:50.445ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:65300 req_id = 7145fea0-0cfd-42fa-8ded-ef5682dd4781 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
128902025-04-30T06:26:50.445ZINFOtest_disk_expunge (dropshot): request completed latency_us = 758 local_addr = [::1]:44559 method = GET remote_addr = [::1]:65300 req_id = 7145fea0-0cfd-42fa-8ded-ef5682dd4781 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
128912025-04-30T06:26:50.445ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128922025-04-30T06:26:50.459ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128932025-04-30T06:26:50.459ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128942025-04-30T06:26:50.459ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
128952025-04-30T06:26:50.459ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128962025-04-30T06:26:50.460ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7145fea0-0cfd-42fa-8ded-ef5682dd4781", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
128972025-04-30T06:26:50.465ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
128982025-04-30T06:26:50.465ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
128992025-04-30T06:26:50.465ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
129002025-04-30T06:26:50.468ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
129012025-04-30T06:26:50.468ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129022025-04-30T06:26:50.468ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129032025-04-30T06:26:50.468ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129042025-04-30T06:26:50.469ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129052025-04-30T06:26:50.476ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129062025-04-30T06:26:50.476ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
129072025-04-30T06:26:50.477ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129082025-04-30T06:26:50.479ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129092025-04-30T06:26:50.491ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
129102025-04-30T06:26:50.492ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129112025-04-30T06:26:50.492ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129122025-04-30T06:26:50.493ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129132025-04-30T06:26:50.541ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
129142025-04-30T06:26:50.541ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
129152025-04-30T06:26:50.541ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 522.736088ms iteration = 46 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
129162025-04-30T06:26:50.541ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 47 reason = Signaled
129172025-04-30T06:26:50.541ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
129182025-04-30T06:26:50.541ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129192025-04-30T06:26:50.542ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129202025-04-30T06:26:50.542ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129212025-04-30T06:26:50.543ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129222025-04-30T06:26:50.545ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
129232025-04-30T06:26:50.547ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129242025-04-30T06:26:50.557ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
129252025-04-30T06:26:50.557ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
129262025-04-30T06:26:50.557ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
129272025-04-30T06:26:50.558ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
129282025-04-30T06:26:50.569ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
129292025-04-30T06:26:50.569ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
129302025-04-30T06:26:50.573ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 03 94 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62563 req_id = 67624d7e-1798-479e-9c44-76037f011f40
129312025-04-30T06:26:50.573ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 916,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62563 req_id = 67624d7e-1798-479e-9c44-76037f011f40
129322025-04-30T06:26:50.574ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
129332025-04-30T06:26:50.574ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129342025-04-30T06:26:50.574ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62563 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 67624d7e-1798-479e-9c44-76037f011f40
129352025-04-30T06:26:50.574ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3275, tv_nsec: 373593100 } })
129362025-04-30T06:26:50.575ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
129372025-04-30T06:26:50.576ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a8 92 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60457 req_id = d54be1f0-ee4e-4adf-8c9e-dbda019f8367
129382025-04-30T06:26:50.576ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43154,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60457 req_id = d54be1f0-ee4e-4adf-8c9e-dbda019f8367
129392025-04-30T06:26:50.577ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129402025-04-30T06:26:50.577ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60457 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d54be1f0-ee4e-4adf-8c9e-dbda019f8367
129412025-04-30T06:26:50.651ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
129422025-04-30T06:26:50.652ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
129432025-04-30T06:26:50.652ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129442025-04-30T06:26:50.652ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129452025-04-30T06:26:50.652ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
129462025-04-30T06:26:50.652ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129472025-04-30T06:26:50.653ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b9 e1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56370 req_id = eb996403-b39e-4bba-9058-1dbc3f8c79fe
129482025-04-30T06:26:50.653ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47585,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56370 req_id = eb996403-b39e-4bba-9058-1dbc3f8c79fe
129492025-04-30T06:26:50.653ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
129502025-04-30T06:26:50.653ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47585, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56370 req_id = eb996403-b39e-4bba-9058-1dbc3f8c79fe
129512025-04-30T06:26:50.655ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
129522025-04-30T06:26:50.655ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
129532025-04-30T06:26:50.655ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
129542025-04-30T06:26:50.655ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129552025-04-30T06:26:50.664ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129562025-04-30T06:26:50.664ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129572025-04-30T06:26:50.664ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
129582025-04-30T06:26:50.664ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129592025-04-30T06:26:50.667ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b33b49f-e5e4-4f35-9be8-6e1b05df7715", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129602025-04-30T06:26:50.667ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
129612025-04-30T06:26:50.669ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "552948c5-3821-41e2-bbf3-60ae84d0e860", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129622025-04-30T06:26:50.669ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
129632025-04-30T06:26:50.670ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129642025-04-30T06:26:50.671ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
129652025-04-30T06:26:50.673ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129662025-04-30T06:26:50.673ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87f79eab-dd5c-4cb0-82ec-32ffd9ad5238", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129672025-04-30T06:26:50.673ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
129682025-04-30T06:26:50.679ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c6ff2f1-c78d-4eb3-b5c5-fb12f5dc635c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129692025-04-30T06:26:50.679ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
129702025-04-30T06:26:50.681ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8a91f4b-f300-493e-a93f-13c06fe2555e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129712025-04-30T06:26:50.682ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
129722025-04-30T06:26:50.684ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e38fd1d-06cf-42c4-a6cd-2c3c3e522fc2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129732025-04-30T06:26:50.684ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
129742025-04-30T06:26:50.687ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b710e421-5a82-4a54-a8c1-31b6b9db3b2d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129752025-04-30T06:26:50.687ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
129762025-04-30T06:26:50.690ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c522c857-772c-4415-be23-d4a2f10ce554", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129772025-04-30T06:26:50.692ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
129782025-04-30T06:26:50.694ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c12ab9f-5dab-4167-b9ab-228c01ea9bf9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129792025-04-30T06:26:50.694ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
129802025-04-30T06:26:50.697ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dbedd83-23ec-4d3d-ac04-6c35273c975a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129812025-04-30T06:26:50.697ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
129822025-04-30T06:26:50.701ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "33709d99-981f-4151-8ec7-89aea1979597", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129832025-04-30T06:26:50.701ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
129842025-04-30T06:26:50.707ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
129852025-04-30T06:26:50.708ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79d4afd4-2a0e-4e93-87de-4a1a7b6a296c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129862025-04-30T06:26:50.708ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
129872025-04-30T06:26:50.711ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "222397d3-a1db-4b76-90fd-118db4d8cfbd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129882025-04-30T06:26:50.711ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
129892025-04-30T06:26:50.714ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57385173-59a8-4bef-88ea-f0b2522ff841", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129902025-04-30T06:26:50.714ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
129912025-04-30T06:26:50.717ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a12b388-e133-4a6e-a8e1-df8abe98c7b6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129922025-04-30T06:26:50.717ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
129932025-04-30T06:26:50.718ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0074d17-fdf1-4f75-a3c3-d661acf6dde1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129942025-04-30T06:26:50.719ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
129952025-04-30T06:26:50.723ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39158512-890c-4565-a8eb-904154c0091f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129962025-04-30T06:26:50.723ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
129972025-04-30T06:26:50.726ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4f11123-8683-4b9b-ad04-e68c8cb39178", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
129982025-04-30T06:26:50.726ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
129992025-04-30T06:26:50.736ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eeebcf2-372e-4d28-8db5-d8750274fd02", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130002025-04-30T06:26:50.736ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
130012025-04-30T06:26:50.737ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63aac1af-8cdc-4b45-99c5-94d38d00b56b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130022025-04-30T06:26:50.737ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
130032025-04-30T06:26:50.739ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f20ab748-7d61-42d4-bc8e-1b5df2054ab7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130042025-04-30T06:26:50.739ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
130052025-04-30T06:26:50.741ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "888c1d96-438c-48e7-8613-74f019b6daa4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130062025-04-30T06:26:50.741ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
130072025-04-30T06:26:50.742ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3baa4ad6-32d6-463a-9862-8c2720ef20ba", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130082025-04-30T06:26:50.742ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
130092025-04-30T06:26:50.744ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a76ee4a3-5820-419c-bf3d-b48acfbc3d2f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130102025-04-30T06:26:50.744ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
130112025-04-30T06:26:50.747ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7093908e-ccb8-4111-9bc0-328dc7af91f8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130122025-04-30T06:26:50.747ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
130132025-04-30T06:26:50.750ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0728f71b-393f-4b09-a2c6-0e471acf4847", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130142025-04-30T06:26:50.750ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
130152025-04-30T06:26:50.753ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "03057232-06b5-46c4-8de2-26cf2e14eef5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130162025-04-30T06:26:50.753ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
130172025-04-30T06:26:50.755ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48fb7ccd-b4e9-409f-abff-c612aba8d6ef", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130182025-04-30T06:26:50.755ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
130192025-04-30T06:26:50.757ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60b0c519-087d-4830-ba2b-a0b548ea6768", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130202025-04-30T06:26:50.757ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
130212025-04-30T06:26:50.758ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130222025-04-30T06:26:50.758ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130232025-04-30T06:26:50.758ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130242025-04-30T06:26:50.758ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
130252025-04-30T06:26:50.760ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130262025-04-30T06:26:50.763ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f045aa9-f182-4c09-8f44-a0acead6cfa8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130272025-04-30T06:26:50.763ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
130282025-04-30T06:26:50.765ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da7dda82-43e1-4af8-a743-5dbaf314993c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130292025-04-30T06:26:50.765ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
130302025-04-30T06:26:50.765ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130312025-04-30T06:26:50.774ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130322025-04-30T06:26:50.774ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130332025-04-30T06:26:50.774ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
130342025-04-30T06:26:50.775ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130352025-04-30T06:26:50.776ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "206e11d9-1793-4239-9551-1bbd44c0407b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130362025-04-30T06:26:50.776ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
130372025-04-30T06:26:50.777ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f14be6f-0a38-4788-8c30-66b6eece81eb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130382025-04-30T06:26:50.777ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
130392025-04-30T06:26:50.778ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130402025-04-30T06:26:50.778ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
130412025-04-30T06:26:50.778ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130422025-04-30T06:26:50.779ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55cd4044-025b-4ed3-a9f5-61732ae318d2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130432025-04-30T06:26:50.779ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
130442025-04-30T06:26:50.781ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b34b1ac6-74ef-407f-8391-b90c2e51f208", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130452025-04-30T06:26:50.781ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
130462025-04-30T06:26:50.784ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51c66fa7-38d9-409e-9f28-795183b3d808", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130472025-04-30T06:26:50.784ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
130482025-04-30T06:26:50.786ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca6f0f25-b5ce-4fbd-8bb9-5abfbc42573d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130492025-04-30T06:26:50.786ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
130502025-04-30T06:26:50.789ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d928f445-9967-4875-9592-31698657dcc6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130512025-04-30T06:26:50.789ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
130522025-04-30T06:26:50.792ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "299e1b4e-3eab-4707-9f6e-69fe47ec02f2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130532025-04-30T06:26:50.792ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
130542025-04-30T06:26:50.794ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf76ffe0-2db8-4017-8ae9-80a21790536c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130552025-04-30T06:26:50.794ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
130562025-04-30T06:26:50.801ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "460dc9c2-ef4d-40c4-a991-2aef2129aa53", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130572025-04-30T06:26:50.802ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
130582025-04-30T06:26:50.804ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aece9292-9cc5-4e65-a7bc-572a3db12111", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130592025-04-30T06:26:50.804ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
130602025-04-30T06:26:50.808ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "927aff54-0120-4ee4-9c4e-fdddb02ac821", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130612025-04-30T06:26:50.808ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
130622025-04-30T06:26:50.809ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "030e564c-a3e1-496d-b8c1-52bf7a039843", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130632025-04-30T06:26:50.809ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
130642025-04-30T06:26:50.818ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1442f7c-778c-40ff-af58-aa3a38cd6a66", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
130652025-04-30T06:26:50.818ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
130662025-04-30T06:26:50.818ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130672025-04-30T06:26:50.819ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130682025-04-30T06:26:50.819ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130692025-04-30T06:26:50.819ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130702025-04-30T06:26:50.820ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
130712025-04-30T06:26:50.822ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130722025-04-30T06:26:50.830ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
130732025-04-30T06:26:50.830ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
130742025-04-30T06:26:50.830ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130752025-04-30T06:26:50.830ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130762025-04-30T06:26:50.830ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130772025-04-30T06:26:50.831ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130782025-04-30T06:26:50.836ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130792025-04-30T06:26:50.845ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
130802025-04-30T06:26:50.846ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130812025-04-30T06:26:50.846ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130822025-04-30T06:26:50.848ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130832025-04-30T06:26:50.916ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
130842025-04-30T06:26:50.916ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
130852025-04-30T06:26:50.916ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130862025-04-30T06:26:50.916ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130872025-04-30T06:26:50.916ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130882025-04-30T06:26:50.916ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
130892025-04-30T06:26:50.917ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130902025-04-30T06:26:50.917ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:38700 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130912025-04-30T06:26:50.917ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
130922025-04-30T06:26:50.918ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130932025-04-30T06:26:50.918ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 486c0293-9ed1-4d77-a580-51496e1beac6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130942025-04-30T06:26:50.919ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:38700 req_id = e5eddaca-7788-4767-aaeb-ad086b9949d3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
130952025-04-30T06:26:50.919ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 486c0293-9ed1-4d77-a580-51496e1beac6 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130962025-04-30T06:26:50.919ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 486c0293-9ed1-4d77-a580-51496e1beac6 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130972025-04-30T06:26:50.919ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 486c0293-9ed1-4d77-a580-51496e1beac6 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
130982025-04-30T06:26:50.920ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1300 local_addr = [::1]:44559 method = GET remote_addr = [::1]:38700 req_id = e5eddaca-7788-4767-aaeb-ad086b9949d3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
130992025-04-30T06:26:50.922ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5eddaca-7788-4767-aaeb-ad086b9949d3", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:49 GMT"} })
131002025-04-30T06:26:50.924ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
131012025-04-30T06:26:50.924ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
131022025-04-30T06:26:50.924ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
131032025-04-30T06:26:50.926ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
131042025-04-30T06:26:50.926ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131052025-04-30T06:26:50.926ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131062025-04-30T06:26:50.926ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131072025-04-30T06:26:50.926ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131082025-04-30T06:26:50.927ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 8692 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 486c0293-9ed1-4d77-a580-51496e1beac6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
131092025-04-30T06:26:50.927ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131102025-04-30T06:26:50.934ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131112025-04-30T06:26:50.934ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131122025-04-30T06:26:50.934ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
131132025-04-30T06:26:50.934ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131142025-04-30T06:26:50.935ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "486c0293-9ed1-4d77-a580-51496e1beac6", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
131152025-04-30T06:26:50.935ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
131162025-04-30T06:26:50.937ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = e2ebbc34-2661-4ae1-8a7e-799c2487f43f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
131172025-04-30T06:26:50.937ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = e2ebbc34-2661-4ae1-8a7e-799c2487f43f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
131182025-04-30T06:26:50.937ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = e2ebbc34-2661-4ae1-8a7e-799c2487f43f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
131192025-04-30T06:26:50.938ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = e2ebbc34-2661-4ae1-8a7e-799c2487f43f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
131202025-04-30T06:26:50.938ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131212025-04-30T06:26:50.943ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
131222025-04-30T06:26:50.943ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131232025-04-30T06:26:50.943ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131242025-04-30T06:26:50.943ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131252025-04-30T06:26:50.944ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 7511 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = e2ebbc34-2661-4ae1-8a7e-799c2487f43f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
131262025-04-30T06:26:50.946ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131272025-04-30T06:26:50.947ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
131282025-04-30T06:26:50.947ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131292025-04-30T06:26:50.948ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2ebbc34-2661-4ae1-8a7e-799c2487f43f", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
131302025-04-30T06:26:50.948ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
131312025-04-30T06:26:50.948ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131322025-04-30T06:26:50.948ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
131332025-04-30T06:26:50.956ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
131342025-04-30T06:26:50.956ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
131352025-04-30T06:26:50.957ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
131362025-04-30T06:26:50.958ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
131372025-04-30T06:26:50.963ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
131382025-04-30T06:26:50.963ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
131392025-04-30T06:26:50.969ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
131402025-04-30T06:26:50.969ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
131412025-04-30T06:26:50.999ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
131422025-04-30T06:26:51.003ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
131432025-04-30T06:26:51.003ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
131442025-04-30T06:26:51.003ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 461.726991ms iteration = 47 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
131452025-04-30T06:26:51.003ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 48 reason = Signaled
131462025-04-30T06:26:51.004ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
131472025-04-30T06:26:51.004ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131482025-04-30T06:26:51.004ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131492025-04-30T06:26:51.004ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131502025-04-30T06:26:51.006ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131512025-04-30T06:26:51.010ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131522025-04-30T06:26:51.017ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
131532025-04-30T06:26:51.017ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131542025-04-30T06:26:51.018ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131552025-04-30T06:26:51.018ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131562025-04-30T06:26:51.023ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
131572025-04-30T06:26:51.023ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
131582025-04-30T06:26:51.028ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 19 a6 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52225 req_id = d3234e4e-faf2-4d53-a368-d11b3cae2666
131592025-04-30T06:26:51.028ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6566,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52225 req_id = d3234e4e-faf2-4d53-a368-d11b3cae2666
131602025-04-30T06:26:51.029ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
131612025-04-30T06:26:51.029ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131622025-04-30T06:26:51.029ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52225 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d3234e4e-faf2-4d53-a368-d11b3cae2666
131632025-04-30T06:26:51.029ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3275, tv_nsec: 828856051 } })
131642025-04-30T06:26:51.030ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
131652025-04-30T06:26:51.038ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d6 0d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55349 req_id = 5033e696-c922-4f14-90ab-e9948f3f7ea7
131662025-04-30T06:26:51.038ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54797,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55349 req_id = 5033e696-c922-4f14-90ab-e9948f3f7ea7
131672025-04-30T06:26:51.039ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131682025-04-30T06:26:51.039ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55349 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5033e696-c922-4f14-90ab-e9948f3f7ea7
131692025-04-30T06:26:51.107ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
131702025-04-30T06:26:51.108ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
131712025-04-30T06:26:51.108ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131722025-04-30T06:26:51.108ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131732025-04-30T06:26:51.108ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
131742025-04-30T06:26:51.110ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131752025-04-30T06:26:51.110ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131762025-04-30T06:26:51.113ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131772025-04-30T06:26:51.113ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
131782025-04-30T06:26:51.115ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 38 f9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 8............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62328 req_id = e343bbca-4b35-4b95-b828-0630a26f99e3
131792025-04-30T06:26:51.115ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14585,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62328 req_id = e343bbca-4b35-4b95-b828-0630a26f99e3
131802025-04-30T06:26:51.115ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
131812025-04-30T06:26:51.115ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 14585, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62328 req_id = e343bbca-4b35-4b95-b828-0630a26f99e3
131822025-04-30T06:26:51.117ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
131832025-04-30T06:26:51.117ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
131842025-04-30T06:26:51.117ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
131852025-04-30T06:26:51.119ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131862025-04-30T06:26:51.120ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131872025-04-30T06:26:51.120ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131882025-04-30T06:26:51.120ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
131892025-04-30T06:26:51.120ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131902025-04-30T06:26:51.127ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131912025-04-30T06:26:51.127ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131922025-04-30T06:26:51.127ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
131932025-04-30T06:26:51.128ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131942025-04-30T06:26:51.137ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131952025-04-30T06:26:51.137ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
131962025-04-30T06:26:51.137ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131972025-04-30T06:26:51.138ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d4d8ab5-d34b-4599-8166-8555f24eaa42", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
131982025-04-30T06:26:51.138ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
131992025-04-30T06:26:51.146ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aeafdcf1-fe98-4508-921a-66e1ceab0a5b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132002025-04-30T06:26:51.146ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
132012025-04-30T06:26:51.154ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae964292-21d3-4100-8001-17c890c89531", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132022025-04-30T06:26:51.154ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
132032025-04-30T06:26:51.157ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e4e03b1-8263-46b6-9f5d-859fbdcdbaae", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132042025-04-30T06:26:51.157ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
132052025-04-30T06:26:51.159ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
132062025-04-30T06:26:51.160ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 07 e1 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:50976 req_id = 4f5fa722-ff8e-44a0-a68a-efbae1b5835b
132072025-04-30T06:26:51.160ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2017,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50976 req_id = 4f5fa722-ff8e-44a0-a68a-efbae1b5835b
132082025-04-30T06:26:51.161ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
132092025-04-30T06:26:51.161ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132102025-04-30T06:26:51.161ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50976 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4f5fa722-ff8e-44a0-a68a-efbae1b5835b
132112025-04-30T06:26:51.163ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3275, tv_nsec: 962213668 } })
132122025-04-30T06:26:51.163ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
132132025-04-30T06:26:51.164ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 61 fc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e a...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57458 req_id = 7272734c-02a3-486e-8b2b-3bb9004f8238
132142025-04-30T06:26:51.164ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25084,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57458 req_id = 7272734c-02a3-486e-8b2b-3bb9004f8238
132152025-04-30T06:26:51.164ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
132162025-04-30T06:26:51.164ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57458 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7272734c-02a3-486e-8b2b-3bb9004f8238
132172025-04-30T06:26:51.165ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
132182025-04-30T06:26:51.222ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
132192025-04-30T06:26:51.222ZDEBGtest_disk_expunge (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
132202025-04-30T06:26:51.222ZDEBGtest_disk_expunge (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
132212025-04-30T06:26:51.222ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
132222025-04-30T06:26:51.224ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
132232025-04-30T06:26:51.225ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "47972102-c44b-41df-9437-51f168e71796", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132242025-04-30T06:26:51.225ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
132252025-04-30T06:26:51.229ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "302fdd31-2cfc-467c-af57-aa79c98263c0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132262025-04-30T06:26:51.229ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
132272025-04-30T06:26:51.233ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed498be2-3ce7-4b72-85da-69d493ec3545", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132282025-04-30T06:26:51.235ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
132292025-04-30T06:26:51.235ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
132302025-04-30T06:26:51.237ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "712dba56-6e55-4a9a-b497-f2dece5b47f3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132312025-04-30T06:26:51.237ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
132322025-04-30T06:26:51.238ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c18f94b-fb89-4693-aac5-e6476f31afc7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132332025-04-30T06:26:51.238ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
132342025-04-30T06:26:51.240ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51a25096-5f5f-457f-bb1f-902eab9cc80c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132352025-04-30T06:26:51.240ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
132362025-04-30T06:26:51.241ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "62439f42-eef3-4405-8172-fae299473f23", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132372025-04-30T06:26:51.241ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
132382025-04-30T06:26:51.242ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2af9926-ff5f-4f54-8393-7262eb983212", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132392025-04-30T06:26:51.242ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
132402025-04-30T06:26:51.243ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b48f0d2-2f02-4736-88e7-bab9323bfc8f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132412025-04-30T06:26:51.243ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
132422025-04-30T06:26:51.245ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83ff7e29-6a79-4ebe-ae71-245ef3c59be9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132432025-04-30T06:26:51.246ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
132442025-04-30T06:26:51.248ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b06b698-dfbc-4f07-9cba-b866689b76d2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132452025-04-30T06:26:51.249ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
132462025-04-30T06:26:51.252ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f62356a8-abe3-4710-854e-dd91444dd618", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132472025-04-30T06:26:51.252ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
132482025-04-30T06:26:51.254ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb256980-f7cf-4321-b307-caea34f1945d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132492025-04-30T06:26:51.255ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
132502025-04-30T06:26:51.258ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2f5dee6-fbfb-472d-b074-ba55037130ef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132512025-04-30T06:26:51.259ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
132522025-04-30T06:26:51.267ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fabc5bee-278a-4e72-a88b-8186cf407293", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132532025-04-30T06:26:51.267ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
132542025-04-30T06:26:51.268ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce9db4f3-9d97-48ff-8e22-eddebeb62a46", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132552025-04-30T06:26:51.268ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
132562025-04-30T06:26:51.275ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "851eb135-7ae9-4b31-98a8-8915801baaf3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132572025-04-30T06:26:51.275ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
132582025-04-30T06:26:51.276ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "418270c8-7ede-4a62-99ee-4f657a059094", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132592025-04-30T06:26:51.276ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
132602025-04-30T06:26:51.277ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fbebf6b-860c-4c36-a5f1-663d97a6d0a8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132612025-04-30T06:26:51.279ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
132622025-04-30T06:26:51.279ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea4fe67c-465e-4fd0-89f4-da73c2d1af12", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132632025-04-30T06:26:51.279ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
132642025-04-30T06:26:51.283ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29a6ee28-aa20-402d-a2f2-a3fef86b3116", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132652025-04-30T06:26:51.284ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
132662025-04-30T06:26:51.286ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
132672025-04-30T06:26:51.286ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132682025-04-30T06:26:51.286ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132692025-04-30T06:26:51.286ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
132702025-04-30T06:26:51.286ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132712025-04-30T06:26:51.287ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af91c336-4cab-47ae-9d3e-6920d593604e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132722025-04-30T06:26:51.287ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
132732025-04-30T06:26:51.292ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132742025-04-30T06:26:51.304ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
132752025-04-30T06:26:51.304ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132762025-04-30T06:26:51.304ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
132772025-04-30T06:26:51.306ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132782025-04-30T06:26:51.308ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132792025-04-30T06:26:51.308ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
132802025-04-30T06:26:51.308ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132812025-04-30T06:26:51.310ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f035518-82b4-48ce-96b5-e7aac0fdb330", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132822025-04-30T06:26:51.310ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
132832025-04-30T06:26:51.317ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b81c9dcc-cc55-4eef-b492-38bdeb6b4bc0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132842025-04-30T06:26:51.317ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
132852025-04-30T06:26:51.322ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4735e331-190c-4fb0-aca7-93cfe4a8df00", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132862025-04-30T06:26:51.322ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
132872025-04-30T06:26:51.327ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b15dc4a-ee01-46c6-b43a-029a15b9064f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132882025-04-30T06:26:51.327ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
132892025-04-30T06:26:51.327ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c14ae231-08cf-43b0-a712-7dc1b0bdf589", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132902025-04-30T06:26:51.327ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
132912025-04-30T06:26:51.328ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cc7b5d2-8569-4313-a952-caf2ba88302b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132922025-04-30T06:26:51.329ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
132932025-04-30T06:26:51.333ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5d29e89-bc0e-48cf-bb86-3ec4cd851683", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132942025-04-30T06:26:51.334ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
132952025-04-30T06:26:51.340ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "831a01d2-5fa6-4b51-8b8d-682be345a44c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132962025-04-30T06:26:51.340ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
132972025-04-30T06:26:51.340ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b1b1fde-9629-4fc6-aea4-daa4d5874a9d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
132982025-04-30T06:26:51.340ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
132992025-04-30T06:26:51.341ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
133002025-04-30T06:26:51.342ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: ad fb 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:62254 req_id = e9c956a7-7d14-446f-ba62-317275adca18
133012025-04-30T06:26:51.342ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44539,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62254 req_id = e9c956a7-7d14-446f-ba62-317275adca18
133022025-04-30T06:26:51.342ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
133032025-04-30T06:26:51.343ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133042025-04-30T06:26:51.343ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62254 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e9c956a7-7d14-446f-ba62-317275adca18
133052025-04-30T06:26:51.347ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3276, tv_nsec: 142946626 } })
133062025-04-30T06:26:51.347ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
133072025-04-30T06:26:51.354ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 67 ed 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e g...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63736 req_id = e097f39c-dbfd-4f1c-8fcf-f4f99147bf92
133082025-04-30T06:26:51.354ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26605,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63736 req_id = e097f39c-dbfd-4f1c-8fcf-f4f99147bf92
133092025-04-30T06:26:51.354ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133102025-04-30T06:26:51.354ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63736 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e097f39c-dbfd-4f1c-8fcf-f4f99147bf92
133112025-04-30T06:26:51.356ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
133122025-04-30T06:26:51.433ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
133132025-04-30T06:26:51.433ZDEBGtest_disk_expunge (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
133142025-04-30T06:26:51.433ZDEBGtest_disk_expunge (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
133152025-04-30T06:26:51.433ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
133162025-04-30T06:26:51.435ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
133172025-04-30T06:26:51.436ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "abbdbe16-639c-4457-bca7-3a8415af74cf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
133182025-04-30T06:26:51.436ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
133192025-04-30T06:26:51.442ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "491ba172-8c10-445a-b601-1f59e8d12547", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
133202025-04-30T06:26:51.442ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
133212025-04-30T06:26:51.443ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
133222025-04-30T06:26:51.459ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b67aeb2e-5a4f-4c17-8291-764219e51eb3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
133232025-04-30T06:26:51.459ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
133242025-04-30T06:26:51.461ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3a49bc0-0632-4d34-a84b-1b9402649302", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
133252025-04-30T06:26:51.461ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
133262025-04-30T06:26:51.463ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5e334b3-bfe0-42b5-b0bf-563f0c48c912", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
133272025-04-30T06:26:51.463ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
133282025-04-30T06:26:51.466ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32863cfe-2721-4638-b12c-1ee7abb2ac94", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
133292025-04-30T06:26:51.466ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
133302025-04-30T06:26:51.470ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1a47b5e-b63f-4273-aedc-4169a9ca544d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
133312025-04-30T06:26:51.470ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
133322025-04-30T06:26:51.473ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30988023-f985-49b9-b52c-0e27fb7d38b5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
133332025-04-30T06:26:51.474ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
133342025-04-30T06:26:51.475ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd1b0682-4365-45a4-aaaa-10e60ca381af", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
133352025-04-30T06:26:51.475ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
133362025-04-30T06:26:51.476ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "074b3cd6-129d-4303-a5e8-42867ed88e00", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
133372025-04-30T06:26:51.476ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
133382025-04-30T06:26:51.476ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133392025-04-30T06:26:51.477ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133402025-04-30T06:26:51.477ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133412025-04-30T06:26:51.477ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133422025-04-30T06:26:51.481ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133432025-04-30T06:26:51.485ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
133442025-04-30T06:26:51.485ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
133452025-04-30T06:26:51.485ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133462025-04-30T06:26:51.485ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133472025-04-30T06:26:51.485ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133482025-04-30T06:26:51.486ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133492025-04-30T06:26:51.493ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133502025-04-30T06:26:51.500ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
133512025-04-30T06:26:51.501ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133522025-04-30T06:26:51.501ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133532025-04-30T06:26:51.501ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133542025-04-30T06:26:51.502ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133552025-04-30T06:26:51.502ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133562025-04-30T06:26:51.502ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133572025-04-30T06:26:51.502ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
133582025-04-30T06:26:51.503ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133592025-04-30T06:26:51.510ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133602025-04-30T06:26:51.518ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133612025-04-30T06:26:51.518ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133622025-04-30T06:26:51.518ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
133632025-04-30T06:26:51.519ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133642025-04-30T06:26:51.600ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
133652025-04-30T06:26:51.601ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
133662025-04-30T06:26:51.601ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:33680 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
133672025-04-30T06:26:51.601ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
133682025-04-30T06:26:51.602ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:33680 req_id = a3ca4708-8b81-4415-aab8-b7987a35e3df sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
133692025-04-30T06:26:51.603ZINFOtest_disk_expunge (dropshot): request completed latency_us = 557 local_addr = [::1]:44559 method = GET remote_addr = [::1]:33680 req_id = a3ca4708-8b81-4415-aab8-b7987a35e3df response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
133702025-04-30T06:26:51.603ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 36 08 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 6............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52735 req_id = c7b68d6a-85ee-4cac-861f-35cdc099d1c6
133712025-04-30T06:26:51.603ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13832,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52735 req_id = c7b68d6a-85ee-4cac-861f-35cdc099d1c6
133722025-04-30T06:26:51.603ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
133732025-04-30T06:26:51.604ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133742025-04-30T06:26:51.604ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52735 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c7b68d6a-85ee-4cac-861f-35cdc099d1c6
133752025-04-30T06:26:51.604ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3ca4708-8b81-4415-aab8-b7987a35e3df", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
133762025-04-30T06:26:51.608ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
133772025-04-30T06:26:51.608ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
133782025-04-30T06:26:51.608ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
133792025-04-30T06:26:51.615ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
133802025-04-30T06:26:51.615ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133812025-04-30T06:26:51.615ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133822025-04-30T06:26:51.615ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133832025-04-30T06:26:51.617ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133842025-04-30T06:26:51.617ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133852025-04-30T06:26:51.617ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
133862025-04-30T06:26:51.618ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133872025-04-30T06:26:51.619ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3276, tv_nsec: 417956095 } })
133882025-04-30T06:26:51.619ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
133892025-04-30T06:26:51.621ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133902025-04-30T06:26:51.624ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
133912025-04-30T06:26:51.625ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133922025-04-30T06:26:51.625ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133932025-04-30T06:26:51.625ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133942025-04-30T06:26:51.626ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 81 b4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53079 req_id = 63883a8a-1b9f-4ecc-a344-17f88c63f153
133952025-04-30T06:26:51.626ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33204,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53079 req_id = 63883a8a-1b9f-4ecc-a344-17f88c63f153
133962025-04-30T06:26:51.627ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133972025-04-30T06:26:51.627ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53079 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 63883a8a-1b9f-4ecc-a344-17f88c63f153
133982025-04-30T06:26:51.627ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
133992025-04-30T06:26:51.679ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
134002025-04-30T06:26:51.679ZDEBGtest_disk_expunge (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
134012025-04-30T06:26:51.680ZDEBGtest_disk_expunge (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
134022025-04-30T06:26:51.680ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
134032025-04-30T06:26:51.682ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
134042025-04-30T06:26:51.730ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
134052025-04-30T06:26:51.731ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
134062025-04-30T06:26:51.732ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 728.335542ms iteration = 48 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
134072025-04-30T06:26:51.732ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 49 reason = Signaled
134082025-04-30T06:26:51.732ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
134092025-04-30T06:26:51.732ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134102025-04-30T06:26:51.732ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134112025-04-30T06:26:51.732ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134122025-04-30T06:26:51.733ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134132025-04-30T06:26:51.735ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
134142025-04-30T06:26:51.739ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134152025-04-30T06:26:51.744ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
134162025-04-30T06:26:51.744ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
134172025-04-30T06:26:51.744ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
134182025-04-30T06:26:51.745ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
134192025-04-30T06:26:51.749ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
134202025-04-30T06:26:51.750ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
134212025-04-30T06:26:51.751ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 7b 6c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 {l..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64904 req_id = 21418b4a-538d-4a7b-980a-eddf66ba0ac7
134222025-04-30T06:26:51.751ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31596,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64904 req_id = 21418b4a-538d-4a7b-980a-eddf66ba0ac7
134232025-04-30T06:26:51.751ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
134242025-04-30T06:26:51.751ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
134252025-04-30T06:26:51.751ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64904 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 21418b4a-538d-4a7b-980a-eddf66ba0ac7
134262025-04-30T06:26:51.752ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3276, tv_nsec: 551365925 } })
134272025-04-30T06:26:51.752ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
134282025-04-30T06:26:51.754ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 69 28 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e i(..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49410 req_id = 83b941d3-482d-4d56-b493-d8644bdd273d
134292025-04-30T06:26:51.754ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26920,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49410 req_id = 83b941d3-482d-4d56-b493-d8644bdd273d
134302025-04-30T06:26:51.754ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
134312025-04-30T06:26:51.754ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49410 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 83b941d3-482d-4d56-b493-d8644bdd273d
134322025-04-30T06:26:51.825ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
134332025-04-30T06:26:51.826ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134342025-04-30T06:26:51.826ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134352025-04-30T06:26:51.826ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134362025-04-30T06:26:51.826ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
134372025-04-30T06:26:51.827ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134382025-04-30T06:26:51.827ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: dd 41 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .A..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54257 req_id = 012836ff-fea1-4a7a-9ef0-3cdbfe22a3e0
134392025-04-30T06:26:51.827ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56641,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54257 req_id = 012836ff-fea1-4a7a-9ef0-3cdbfe22a3e0
134402025-04-30T06:26:51.827ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
134412025-04-30T06:26:51.827ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56641, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54257 req_id = 012836ff-fea1-4a7a-9ef0-3cdbfe22a3e0
134422025-04-30T06:26:51.828ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
134432025-04-30T06:26:51.828ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
134442025-04-30T06:26:51.828ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
134452025-04-30T06:26:51.829ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134462025-04-30T06:26:51.836ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134472025-04-30T06:26:51.836ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134482025-04-30T06:26:51.837ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
134492025-04-30T06:26:51.837ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134502025-04-30T06:26:51.838ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfb33484-48fb-4be8-8b8f-1e2b6370eca3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134512025-04-30T06:26:51.839ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
134522025-04-30T06:26:51.841ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134532025-04-30T06:26:51.841ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
134542025-04-30T06:26:51.842ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134552025-04-30T06:26:51.844ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b055937-f4d5-4228-b324-19cbf58d31bd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134562025-04-30T06:26:51.845ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
134572025-04-30T06:26:51.853ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f166854-07e4-4821-8c3e-9910b6aa4d67", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134582025-04-30T06:26:51.854ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
134592025-04-30T06:26:51.862ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52d87d86-7666-4fcb-bf82-4be8794232ae", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134602025-04-30T06:26:51.862ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
134612025-04-30T06:26:51.868ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20ed8347-1629-421a-b379-35924b792cef", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134622025-04-30T06:26:51.868ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
134632025-04-30T06:26:51.871ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4d5c784-2415-4ea8-9447-49ec6ca6f11c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134642025-04-30T06:26:51.871ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
134652025-04-30T06:26:51.874ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e4138aa-344b-415f-ac63-0e3890dc62f9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134662025-04-30T06:26:51.874ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
134672025-04-30T06:26:51.877ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f50a992-bd2e-4ed7-ae2c-fff357dbcdbe", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134682025-04-30T06:26:51.878ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
134692025-04-30T06:26:51.880ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92bcaa7f-9178-451d-b462-83de0918c620", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134702025-04-30T06:26:51.880ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
134712025-04-30T06:26:51.885ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
134722025-04-30T06:26:51.886ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "99e4172c-3079-485a-8a62-0e0f49372832", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134732025-04-30T06:26:51.886ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
134742025-04-30T06:26:51.891ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05fb864c-cfd2-46c6-a631-137243581dd9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134752025-04-30T06:26:51.892ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
134762025-04-30T06:26:51.894ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ade0a373-47e4-4254-9d97-7fafe831c758", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134772025-04-30T06:26:51.894ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
134782025-04-30T06:26:51.896ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65c9d7b1-8717-40ba-9f90-ffe6a24a0297", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134792025-04-30T06:26:51.896ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
134802025-04-30T06:26:51.899ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a83667ef-98cc-4d31-addb-9df73f70bb57", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134812025-04-30T06:26:51.900ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
134822025-04-30T06:26:51.903ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7609f63-7f4c-42e7-bf3a-680f8938df13", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134832025-04-30T06:26:51.903ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
134842025-04-30T06:26:51.907ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79b8d238-83a4-4a5a-9788-2f6bb9247747", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134852025-04-30T06:26:51.907ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
134862025-04-30T06:26:51.910ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56ab5f31-42d4-4ac7-9627-2d1082ef3113", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134872025-04-30T06:26:51.910ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
134882025-04-30T06:26:51.913ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25db3a8f-49c2-492c-8fd1-7574c45cee91", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134892025-04-30T06:26:51.914ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
134902025-04-30T06:26:51.917ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0496edac-6eae-44ef-a2f9-8e061959c7b7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134912025-04-30T06:26:51.917ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
134922025-04-30T06:26:51.918ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2856fadb-20e5-43ef-9b47-3c205fa9d8e7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134932025-04-30T06:26:51.919ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
134942025-04-30T06:26:51.920ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03f87132-6eab-456f-87a6-4f7a0affc395", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134952025-04-30T06:26:51.920ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
134962025-04-30T06:26:51.922ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12b534ec-dd9e-4291-a36e-8a411575c902", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134972025-04-30T06:26:51.922ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
134982025-04-30T06:26:51.923ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "561b56b0-521c-4822-b9fa-3e1823c9764f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
134992025-04-30T06:26:51.923ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
135002025-04-30T06:26:51.925ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75bfddd4-56b3-48a1-bb5c-e7148f67e8cd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:50 GMT"} })
135012025-04-30T06:26:51.926ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
135022025-04-30T06:26:51.929ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7fe9fbf-a633-47a5-92ac-f0da66c08950", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135032025-04-30T06:26:51.929ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
135042025-04-30T06:26:51.932ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46e8daca-f762-4f9a-82f0-8793709e9d5c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135052025-04-30T06:26:51.932ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
135062025-04-30T06:26:51.936ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e823f713-f79d-4bb7-ad84-e1485d48c2d4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135072025-04-30T06:26:51.936ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
135082025-04-30T06:26:51.937ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135092025-04-30T06:26:51.937ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135102025-04-30T06:26:51.937ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135112025-04-30T06:26:51.937ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
135122025-04-30T06:26:51.938ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135132025-04-30T06:26:51.940ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135142025-04-30T06:26:51.949ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135152025-04-30T06:26:51.949ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135162025-04-30T06:26:51.949ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
135172025-04-30T06:26:51.950ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135182025-04-30T06:26:51.958ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
135192025-04-30T06:26:51.958ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
135202025-04-30T06:26:51.959ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
135212025-04-30T06:26:51.960ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
135222025-04-30T06:26:51.960ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135232025-04-30T06:26:51.960ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
135242025-04-30T06:26:51.961ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135252025-04-30T06:26:51.963ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f353f9ee-026d-4213-94cc-cbb81017b349", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135262025-04-30T06:26:51.966ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
135272025-04-30T06:26:51.967ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
135282025-04-30T06:26:51.968ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
135292025-04-30T06:26:51.972ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
135302025-04-30T06:26:51.972ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
135312025-04-30T06:26:51.973ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be896532-c43c-4922-aa2d-ad90f699ab1a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135322025-04-30T06:26:51.974ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
135332025-04-30T06:26:51.990ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "751cdd0d-7ef8-4996-baab-8d3748d470e8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135342025-04-30T06:26:51.990ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
135352025-04-30T06:26:51.995ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91a67bb9-842b-4299-8ded-00ae88f0f020", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135362025-04-30T06:26:51.995ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
135372025-04-30T06:26:51.998ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90a15a48-d7f8-4318-a89d-ef58fd657f07", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135382025-04-30T06:26:51.999ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
135392025-04-30T06:26:52.003ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce2c9b6d-1d21-4340-b6d1-4cee6c160967", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135402025-04-30T06:26:52.003ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
135412025-04-30T06:26:52.005ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b347cf6-9002-469a-a7d9-02f0f33df13b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135422025-04-30T06:26:52.005ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
135432025-04-30T06:26:52.008ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aa298d6-ead0-4b99-9064-0146a4d3ed5d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135442025-04-30T06:26:52.009ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
135452025-04-30T06:26:52.013ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
135462025-04-30T06:26:52.048ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135472025-04-30T06:26:52.049ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135482025-04-30T06:26:52.057ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135492025-04-30T06:26:52.064ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135502025-04-30T06:26:52.064ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135512025-04-30T06:26:52.064ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135522025-04-30T06:26:52.065ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
135532025-04-30T06:26:52.065ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135542025-04-30T06:26:52.066ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135552025-04-30T06:26:52.066ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135562025-04-30T06:26:52.066ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
135572025-04-30T06:26:52.066ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135582025-04-30T06:26:52.072ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135592025-04-30T06:26:52.080ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135602025-04-30T06:26:52.080ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135612025-04-30T06:26:52.080ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
135622025-04-30T06:26:52.081ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135632025-04-30T06:26:52.087ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32608858-c581-42a2-aa49-a2e9b17d0aa7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135642025-04-30T06:26:52.087ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
135652025-04-30T06:26:52.087ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135662025-04-30T06:26:52.087ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
135672025-04-30T06:26:52.088ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135682025-04-30T06:26:52.130ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b559b152-3798-4c70-9dc2-00d9123ba640", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135692025-04-30T06:26:52.130ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
135702025-04-30T06:26:52.132ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c48345c0-9125-463c-b8ee-26953bb06822", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135712025-04-30T06:26:52.133ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
135722025-04-30T06:26:52.135ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55d720e5-1f69-45bd-ba5c-c390d0845f94", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135732025-04-30T06:26:52.136ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
135742025-04-30T06:26:52.140ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b83b09c-2eb9-4c1c-a4d4-480528151987", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135752025-04-30T06:26:52.140ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
135762025-04-30T06:26:52.145ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9053e4d6-7200-4b28-8331-856b2848cf72", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135772025-04-30T06:26:52.145ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
135782025-04-30T06:26:52.149ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60460242-3642-45f6-9840-7d60a0dffb37", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135792025-04-30T06:26:52.149ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
135802025-04-30T06:26:52.150ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
135812025-04-30T06:26:52.152ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d4fe6bd-b273-4f4a-8a0c-5fda5d5eedde", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135822025-04-30T06:26:52.153ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
135832025-04-30T06:26:52.155ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "431fb69c-717f-49ca-932b-d6faef5cc4d5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135842025-04-30T06:26:52.155ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
135852025-04-30T06:26:52.157ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84c6cf59-83df-406a-adbe-38bedaa43cd1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
135862025-04-30T06:26:52.157ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
135872025-04-30T06:26:52.157ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135882025-04-30T06:26:52.157ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135892025-04-30T06:26:52.157ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135902025-04-30T06:26:52.158ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135912025-04-30T06:26:52.178ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135922025-04-30T06:26:52.183ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
135932025-04-30T06:26:52.183ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
135942025-04-30T06:26:52.183ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135952025-04-30T06:26:52.183ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135962025-04-30T06:26:52.183ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135972025-04-30T06:26:52.185ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135982025-04-30T06:26:52.188ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135992025-04-30T06:26:52.195ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
136002025-04-30T06:26:52.195ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136012025-04-30T06:26:52.195ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136022025-04-30T06:26:52.197ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136032025-04-30T06:26:52.202ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
136042025-04-30T06:26:52.202ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136052025-04-30T06:26:52.202ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136062025-04-30T06:26:52.202ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
136072025-04-30T06:26:52.203ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136082025-04-30T06:26:52.265ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
136092025-04-30T06:26:52.265ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
136102025-04-30T06:26:52.265ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:46387 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
136112025-04-30T06:26:52.266ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:46387 req_id = 2726d1bf-69c6-419a-bb1d-c464bf15aa7e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
136122025-04-30T06:26:52.267ZINFOtest_disk_expunge (dropshot): request completed latency_us = 607 local_addr = [::1]:44559 method = GET remote_addr = [::1]:46387 req_id = 2726d1bf-69c6-419a-bb1d-c464bf15aa7e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
136132025-04-30T06:26:52.267ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136142025-04-30T06:26:52.276ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
136152025-04-30T06:26:52.276ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136162025-04-30T06:26:52.276ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
136172025-04-30T06:26:52.277ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136182025-04-30T06:26:52.277ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2726d1bf-69c6-419a-bb1d-c464bf15aa7e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
136192025-04-30T06:26:52.279ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
136202025-04-30T06:26:52.279ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
136212025-04-30T06:26:52.279ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
136222025-04-30T06:26:52.281ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
136232025-04-30T06:26:52.281ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136242025-04-30T06:26:52.281ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136252025-04-30T06:26:52.282ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136262025-04-30T06:26:52.283ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136272025-04-30T06:26:52.312ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136282025-04-30T06:26:52.312ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
136292025-04-30T06:26:52.313ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136302025-04-30T06:26:52.314ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136312025-04-30T06:26:52.321ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
136322025-04-30T06:26:52.321ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136332025-04-30T06:26:52.321ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136342025-04-30T06:26:52.322ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136352025-04-30T06:26:52.361ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
136362025-04-30T06:26:52.363ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
136372025-04-30T06:26:52.364ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
136382025-04-30T06:26:52.364ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 631.943616ms iteration = 49 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
136392025-04-30T06:26:52.364ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 50 reason = Signaled
136402025-04-30T06:26:52.364ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
136412025-04-30T06:26:52.364ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136422025-04-30T06:26:52.364ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136432025-04-30T06:26:52.364ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136442025-04-30T06:26:52.364ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136452025-04-30T06:26:52.372ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136462025-04-30T06:26:52.385ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
136472025-04-30T06:26:52.385ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136482025-04-30T06:26:52.385ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136492025-04-30T06:26:52.386ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136502025-04-30T06:26:52.390ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
136512025-04-30T06:26:52.390ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
136522025-04-30T06:26:52.391ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 50 a9 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 P............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51510 req_id = a0e8b905-e7d5-4f7b-8fae-f430322ae5f9
136532025-04-30T06:26:52.391ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20649,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51510 req_id = a0e8b905-e7d5-4f7b-8fae-f430322ae5f9
136542025-04-30T06:26:52.391ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
136552025-04-30T06:26:52.392ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136562025-04-30T06:26:52.392ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51510 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a0e8b905-e7d5-4f7b-8fae-f430322ae5f9
136572025-04-30T06:26:52.392ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3277, tv_nsec: 191726883 } })
136582025-04-30T06:26:52.392ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
136592025-04-30T06:26:52.393ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 82 b1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58512 req_id = aa4d5b3e-6af9-4112-94aa-940aa94710d3
136602025-04-30T06:26:52.393ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33457,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58512 req_id = aa4d5b3e-6af9-4112-94aa-940aa94710d3
136612025-04-30T06:26:52.393ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136622025-04-30T06:26:52.393ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58512 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = aa4d5b3e-6af9-4112-94aa-940aa94710d3
136632025-04-30T06:26:52.452ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
136642025-04-30T06:26:52.452ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
136652025-04-30T06:26:52.452ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136662025-04-30T06:26:52.452ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136672025-04-30T06:26:52.452ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
136682025-04-30T06:26:52.452ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136692025-04-30T06:26:52.453ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: bf 6a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .j..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58291 req_id = 0f1e3e8e-5035-4897-bae1-77c4b666d3db
136702025-04-30T06:26:52.453ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49002,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58291 req_id = 0f1e3e8e-5035-4897-bae1-77c4b666d3db
136712025-04-30T06:26:52.453ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
136722025-04-30T06:26:52.453ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49002, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58291 req_id = 0f1e3e8e-5035-4897-bae1-77c4b666d3db
136732025-04-30T06:26:52.454ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
136742025-04-30T06:26:52.454ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
136752025-04-30T06:26:52.454ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
136762025-04-30T06:26:52.456ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136772025-04-30T06:26:52.471ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
136782025-04-30T06:26:52.471ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136792025-04-30T06:26:52.471ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
136802025-04-30T06:26:52.473ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136812025-04-30T06:26:52.478ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "da2cd75b-cb7f-43cf-8c50-5f05031bcb95", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
136822025-04-30T06:26:52.478ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
136832025-04-30T06:26:52.479ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136842025-04-30T06:26:52.479ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
136852025-04-30T06:26:52.481ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136862025-04-30T06:26:52.484ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e11f4b8-744e-4f5d-886a-64e39329f67d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
136872025-04-30T06:26:52.485ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
136882025-04-30T06:26:52.495ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ac715fe-239e-42ae-9c64-d9de21e3a04c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
136892025-04-30T06:26:52.496ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
136902025-04-30T06:26:52.502ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "310e4164-c5e9-4d1d-b5a1-83f61317f26f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
136912025-04-30T06:26:52.502ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
136922025-04-30T06:26:52.508ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfca0d38-f543-4ea3-b0a0-e4029b898f24", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
136932025-04-30T06:26:52.508ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
136942025-04-30T06:26:52.514ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c406d3f-4141-48cb-b507-b43180b3be0c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
136952025-04-30T06:26:52.514ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
136962025-04-30T06:26:52.526ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0fb5ff4-4588-4596-9208-8eacbfac0ac1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
136972025-04-30T06:26:52.526ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
136982025-04-30T06:26:52.536ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
136992025-04-30T06:26:52.536ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bbfbdd6-b548-42be-8c06-f28ebd7367c2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137002025-04-30T06:26:52.537ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
137012025-04-30T06:26:52.539ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ad36aed-88a5-4498-816a-e3857f1b5ca6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137022025-04-30T06:26:52.539ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
137032025-04-30T06:26:52.542ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f789f7b-842a-49db-b6a7-da5bcba43566", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137042025-04-30T06:26:52.542ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
137052025-04-30T06:26:52.544ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e566248d-ecbf-45d9-ad44-f7096f0d9794", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137062025-04-30T06:26:52.544ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
137072025-04-30T06:26:52.546ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e2d7342-b8ec-4e3b-89f5-33b00f5dcb31", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137082025-04-30T06:26:52.547ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
137092025-04-30T06:26:52.549ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d29037c6-04eb-4246-a7f3-458b9a1277c2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137102025-04-30T06:26:52.549ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
137112025-04-30T06:26:52.552ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88dcf4d1-bd4b-46aa-9d33-910da1cfcd62", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137122025-04-30T06:26:52.552ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
137132025-04-30T06:26:52.555ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7dad60e-5bd1-4d94-aefe-bb57438e52bb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137142025-04-30T06:26:52.555ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
137152025-04-30T06:26:52.558ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18e92bac-cdd4-43c9-a0ce-1c3b37dd95c4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137162025-04-30T06:26:52.558ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
137172025-04-30T06:26:52.561ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "890e69c8-8d3b-4591-83f3-ae4e310b00c0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137182025-04-30T06:26:52.561ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
137192025-04-30T06:26:52.563ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d48bf98-4501-454a-9372-2536d420f859", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137202025-04-30T06:26:52.564ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
137212025-04-30T06:26:52.566ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9f1f259-bcaa-4bd8-b57f-fc2c3071187e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137222025-04-30T06:26:52.566ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
137232025-04-30T06:26:52.567ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b347d890-e73f-497e-9d74-d8581b554adc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137242025-04-30T06:26:52.567ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
137252025-04-30T06:26:52.568ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "583c27b0-93d6-4d56-868c-8247b25c5a7c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137262025-04-30T06:26:52.568ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
137272025-04-30T06:26:52.571ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a61f057-c65a-492d-832a-a2a28d5d6e64", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137282025-04-30T06:26:52.572ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
137292025-04-30T06:26:52.573ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27948949-2bb5-4712-a48c-187fe5115c2b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137302025-04-30T06:26:52.573ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
137312025-04-30T06:26:52.577ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "548550db-1ec8-4a68-9147-e64cb83926dd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137322025-04-30T06:26:52.578ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
137332025-04-30T06:26:52.580ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c9f3d5a-a917-4711-98bc-2e350ec9383d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137342025-04-30T06:26:52.580ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
137352025-04-30T06:26:52.584ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9635f5bd-38bc-4c1e-958d-020f72a12860", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137362025-04-30T06:26:52.585ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
137372025-04-30T06:26:52.586ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
137382025-04-30T06:26:52.587ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137392025-04-30T06:26:52.587ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137402025-04-30T06:26:52.587ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
137412025-04-30T06:26:52.598ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137422025-04-30T06:26:52.601ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "678a6f51-8173-4c27-9d3f-71980f103655", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137432025-04-30T06:26:52.601ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
137442025-04-30T06:26:52.602ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137452025-04-30T06:26:52.614ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
137462025-04-30T06:26:52.614ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137472025-04-30T06:26:52.614ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
137482025-04-30T06:26:52.616ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137492025-04-30T06:26:52.621ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137502025-04-30T06:26:52.621ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
137512025-04-30T06:26:52.621ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137522025-04-30T06:26:52.624ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53d4aa56-29e6-47c9-b4b6-a62bfa223d71", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137532025-04-30T06:26:52.624ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
137542025-04-30T06:26:52.630ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cd34f97-70a6-4594-9758-72398034c65a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137552025-04-30T06:26:52.630ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
137562025-04-30T06:26:52.637ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb3892cd-b295-4463-a47c-c811b77026e7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137572025-04-30T06:26:52.637ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
137582025-04-30T06:26:52.641ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57af4eb0-7d98-411c-b8cd-b38e11ff3a2b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137592025-04-30T06:26:52.641ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
137602025-04-30T06:26:52.643ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "367877b6-f2ed-4bd2-83ee-29bce09e3f00", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137612025-04-30T06:26:52.643ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
137622025-04-30T06:26:52.646ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "baf42716-d8fe-4d02-be5f-ca56959518ed", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137632025-04-30T06:26:52.646ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
137642025-04-30T06:26:52.648ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7405855-b3c8-47d1-b898-5fbc9b680c34", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137652025-04-30T06:26:52.648ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
137662025-04-30T06:26:52.651ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34274fcc-7c28-4a30-935e-81ac3bd6e5f6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137672025-04-30T06:26:52.651ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
137682025-04-30T06:26:52.653ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f672763-bbdf-4701-a856-20db3323fafa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137692025-04-30T06:26:52.653ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
137702025-04-30T06:26:52.656ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
137712025-04-30T06:26:52.659ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d00c460-1571-43a5-9f2f-2103f4c1927b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137722025-04-30T06:26:52.659ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
137732025-04-30T06:26:52.661ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b8035b5-7523-4e76-8f05-1628435f4a16", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137742025-04-30T06:26:52.661ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
137752025-04-30T06:26:52.663ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd1d5ec6-72f0-4381-a398-eb03d86725d9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137762025-04-30T06:26:52.663ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
137772025-04-30T06:26:52.666ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5230e8c-602b-4f0d-9798-4b333d14e5c9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137782025-04-30T06:26:52.666ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
137792025-04-30T06:26:52.669ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6edbc1ea-fb20-4b00-b77f-294735a18689", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137802025-04-30T06:26:52.669ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
137812025-04-30T06:26:52.672ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e3f8887-8fad-4bfa-a8a8-5a321e6af468", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137822025-04-30T06:26:52.673ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
137832025-04-30T06:26:52.676ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84b7c394-8f7a-4f7d-b57e-b10044e31a95", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137842025-04-30T06:26:52.677ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
137852025-04-30T06:26:52.678ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d25dbb06-8d4b-428c-b5a8-164be8abe43b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137862025-04-30T06:26:52.679ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
137872025-04-30T06:26:52.680ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72cfef7d-2e42-4936-b350-a5e8674d526d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
137882025-04-30T06:26:52.680ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
137892025-04-30T06:26:52.681ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137902025-04-30T06:26:52.681ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137912025-04-30T06:26:52.681ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137922025-04-30T06:26:52.682ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137932025-04-30T06:26:52.685ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137942025-04-30T06:26:52.696ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
137952025-04-30T06:26:52.696ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
137962025-04-30T06:26:52.696ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137972025-04-30T06:26:52.696ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137982025-04-30T06:26:52.696ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137992025-04-30T06:26:52.697ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138002025-04-30T06:26:52.702ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138012025-04-30T06:26:52.708ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
138022025-04-30T06:26:52.709ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138032025-04-30T06:26:52.709ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138042025-04-30T06:26:52.711ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138052025-04-30T06:26:52.711ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
138062025-04-30T06:26:52.711ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138072025-04-30T06:26:52.711ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138082025-04-30T06:26:52.712ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
138092025-04-30T06:26:52.713ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138102025-04-30T06:26:52.766ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
138112025-04-30T06:26:52.766ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
138122025-04-30T06:26:52.767ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138132025-04-30T06:26:52.776ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
138142025-04-30T06:26:52.776ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138152025-04-30T06:26:52.776ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
138162025-04-30T06:26:52.776ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138172025-04-30T06:26:52.776ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:39939 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
138182025-04-30T06:26:52.779ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:39939 req_id = f80b48f6-986d-4411-b85c-225b388d70eb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
138192025-04-30T06:26:52.779ZINFOtest_disk_expunge (dropshot): request completed latency_us = 580 local_addr = [::1]:44559 method = GET remote_addr = [::1]:39939 req_id = f80b48f6-986d-4411-b85c-225b388d70eb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
138202025-04-30T06:26:52.779ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f80b48f6-986d-4411-b85c-225b388d70eb", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
138212025-04-30T06:26:52.785ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
138222025-04-30T06:26:52.785ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
138232025-04-30T06:26:52.786ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
138242025-04-30T06:26:52.789ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
138252025-04-30T06:26:52.789ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138262025-04-30T06:26:52.789ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138272025-04-30T06:26:52.789ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138282025-04-30T06:26:52.790ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138292025-04-30T06:26:52.790ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138302025-04-30T06:26:52.790ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
138312025-04-30T06:26:52.791ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138322025-04-30T06:26:52.794ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138332025-04-30T06:26:52.803ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
138342025-04-30T06:26:52.804ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138352025-04-30T06:26:52.804ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138362025-04-30T06:26:52.805ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138372025-04-30T06:26:52.846ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
138382025-04-30T06:26:52.849ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
138392025-04-30T06:26:52.849ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
138402025-04-30T06:26:52.849ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 485.404818ms iteration = 50 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
138412025-04-30T06:26:52.849ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 51 reason = Signaled
138422025-04-30T06:26:52.849ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
138432025-04-30T06:26:52.849ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138442025-04-30T06:26:52.849ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138452025-04-30T06:26:52.849ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138462025-04-30T06:26:52.851ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138472025-04-30T06:26:52.854ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138482025-04-30T06:26:52.868ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
138492025-04-30T06:26:52.868ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138502025-04-30T06:26:52.868ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138512025-04-30T06:26:52.870ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138522025-04-30T06:26:52.875ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
138532025-04-30T06:26:52.875ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138542025-04-30T06:26:52.875ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 46205e50-e0fe-490e-ba61-ff5440adc4ae uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138552025-04-30T06:26:52.876ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 46205e50-e0fe-490e-ba61-ff5440adc4ae resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138562025-04-30T06:26:52.876ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 46205e50-e0fe-490e-ba61-ff5440adc4ae roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138572025-04-30T06:26:52.877ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 46205e50-e0fe-490e-ba61-ff5440adc4ae resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138582025-04-30T06:26:52.877ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
138592025-04-30T06:26:52.877ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
138602025-04-30T06:26:52.879ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 82 0d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53556 req_id = a7e1159a-3597-4bc9-a04d-ee7e7e75cf0f
138612025-04-30T06:26:52.879ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33293,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53556 req_id = a7e1159a-3597-4bc9-a04d-ee7e7e75cf0f
138622025-04-30T06:26:52.879ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
138632025-04-30T06:26:52.880ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138642025-04-30T06:26:52.880ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53556 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a7e1159a-3597-4bc9-a04d-ee7e7e75cf0f
138652025-04-30T06:26:52.880ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 4582 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = 46205e50-e0fe-490e-ba61-ff5440adc4ae response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
138662025-04-30T06:26:52.880ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3277, tv_nsec: 679876318 } })
138672025-04-30T06:26:52.881ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
138682025-04-30T06:26:52.881ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "46205e50-e0fe-490e-ba61-ff5440adc4ae", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:51 GMT"} })
138692025-04-30T06:26:52.881ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138702025-04-30T06:26:52.882ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f030df31-347f-4f6e-b729-2c97b111abaf uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138712025-04-30T06:26:52.882ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f030df31-347f-4f6e-b729-2c97b111abaf resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138722025-04-30T06:26:52.882ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f030df31-347f-4f6e-b729-2c97b111abaf roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138732025-04-30T06:26:52.882ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f030df31-347f-4f6e-b729-2c97b111abaf resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138742025-04-30T06:26:52.883ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d6 9c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52389 req_id = 98e05918-e2a1-4040-b880-8fbd3e367adf
138752025-04-30T06:26:52.883ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54940,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52389 req_id = 98e05918-e2a1-4040-b880-8fbd3e367adf
138762025-04-30T06:26:52.883ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138772025-04-30T06:26:52.883ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52389 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 98e05918-e2a1-4040-b880-8fbd3e367adf
138782025-04-30T06:26:52.946ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
138792025-04-30T06:26:52.947ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 65197 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f030df31-347f-4f6e-b729-2c97b111abaf response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
138802025-04-30T06:26:52.947ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
138812025-04-30T06:26:52.947ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138822025-04-30T06:26:52.947ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138832025-04-30T06:26:52.947ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
138842025-04-30T06:26:52.948ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138852025-04-30T06:26:52.949ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "f030df31-347f-4f6e-b729-2c97b111abaf", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
138862025-04-30T06:26:52.949ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
138872025-04-30T06:26:52.949ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
138882025-04-30T06:26:52.949ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
138892025-04-30T06:26:52.950ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b7 9e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58055 req_id = 4aedf7df-b84e-4f3e-8be6-4aa82b3f4843
138902025-04-30T06:26:52.950ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47006,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58055 req_id = 4aedf7df-b84e-4f3e-8be6-4aa82b3f4843
138912025-04-30T06:26:52.951ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
138922025-04-30T06:26:52.951ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47006, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58055 req_id = 4aedf7df-b84e-4f3e-8be6-4aa82b3f4843
138932025-04-30T06:26:52.952ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
138942025-04-30T06:26:52.953ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
138952025-04-30T06:26:52.953ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
138962025-04-30T06:26:52.954ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138972025-04-30T06:26:52.961ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
138982025-04-30T06:26:52.961ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138992025-04-30T06:26:52.961ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
139002025-04-30T06:26:52.961ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139012025-04-30T06:26:52.969ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
139022025-04-30T06:26:52.969ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
139032025-04-30T06:26:52.969ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
139042025-04-30T06:26:52.969ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
139052025-04-30T06:26:52.969ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139062025-04-30T06:26:52.969ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
139072025-04-30T06:26:52.969ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139082025-04-30T06:26:52.972ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
139092025-04-30T06:26:52.972ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
139102025-04-30T06:26:52.972ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "dae5f63a-0268-42ed-a816-4261400a0644", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139112025-04-30T06:26:52.972ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
139122025-04-30T06:26:52.976ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
139132025-04-30T06:26:52.976ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
139142025-04-30T06:26:52.976ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f55cf6d9-277e-4905-8512-5cc01e6d8df0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139152025-04-30T06:26:52.977ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
139162025-04-30T06:26:52.980ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddcc98aa-7d7b-476d-a32d-2e6005d553c3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139172025-04-30T06:26:52.981ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
139182025-04-30T06:26:52.987ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4af5365-45e2-41c5-bdf8-a4eaafaa2887", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139192025-04-30T06:26:52.987ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
139202025-04-30T06:26:52.990ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5c782fd-6b0d-4af2-b57d-06675c3aea09", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139212025-04-30T06:26:52.990ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
139222025-04-30T06:26:52.992ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cf0a29f-49f9-457b-8279-12e430300fe2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139232025-04-30T06:26:52.992ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
139242025-04-30T06:26:52.995ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "03cf025f-f9d4-4c80-98d8-88f020611f2f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139252025-04-30T06:26:52.995ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
139262025-04-30T06:26:52.998ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fb9bb9e-a07c-46a9-9df0-c90de08d2878", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139272025-04-30T06:26:52.998ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
139282025-04-30T06:26:53.001ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e108d30b-7fb1-42b9-8b6e-28420b5dfffc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139292025-04-30T06:26:53.001ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
139302025-04-30T06:26:53.002ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
139312025-04-30T06:26:53.003ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a1033d2-a037-4beb-9246-0e949f70fc44", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139322025-04-30T06:26:53.003ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
139332025-04-30T06:26:53.006ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d3318a8-5062-4432-a17c-c91a1919f236", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139342025-04-30T06:26:53.006ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
139352025-04-30T06:26:53.008ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a8a6ecc-8cb9-454a-a00f-0305fd4febfb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139362025-04-30T06:26:53.009ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
139372025-04-30T06:26:53.010ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a64ed7d-4710-4f1b-a4e5-d2dc77f049fa", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139382025-04-30T06:26:53.010ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
139392025-04-30T06:26:53.013ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b3e13c2-4fca-4d7d-98be-5c3db141f601", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139402025-04-30T06:26:53.013ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
139412025-04-30T06:26:53.016ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae7477cc-5fbb-4e9b-b368-59db5e4b9b25", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139422025-04-30T06:26:53.016ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
139432025-04-30T06:26:53.044ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139442025-04-30T06:26:53.044ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139452025-04-30T06:26:53.045ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:51524 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
139462025-04-30T06:26:53.048ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:51524 req_id = 5cd5f7a8-8a57-464c-97c4-b92f379895db sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139472025-04-30T06:26:53.048ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139482025-04-30T06:26:53.049ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139492025-04-30T06:26:53.049ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139502025-04-30T06:26:53.049ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139512025-04-30T06:26:53.049ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 1150 local_addr = [::1]:47737 method = GET remote_addr = [::1]:51524 req_id = 5cd5f7a8-8a57-464c-97c4-b92f379895db response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139522025-04-30T06:26:53.054ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
139532025-04-30T06:26:53.054ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139542025-04-30T06:26:53.054ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139552025-04-30T06:26:53.054ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
139562025-04-30T06:26:53.054ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139572025-04-30T06:26:53.066ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e010851-fe07-44bf-b0c8-ff1b68ecce9e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139582025-04-30T06:26:53.067ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
139592025-04-30T06:26:53.067ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139602025-04-30T06:26:53.075ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
139612025-04-30T06:26:53.075ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139622025-04-30T06:26:53.077ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
139632025-04-30T06:26:53.078ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139642025-04-30T06:26:53.080ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
139652025-04-30T06:26:53.087ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139662025-04-30T06:26:53.087ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
139672025-04-30T06:26:53.088ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139682025-04-30T06:26:53.097ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42bc9887-f0f0-4bf6-a572-ce2a51d080e9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139692025-04-30T06:26:53.097ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
139702025-04-30T06:26:53.101ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32cff76f-f313-411b-bb0f-5e10b9c2de2c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139712025-04-30T06:26:53.101ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
139722025-04-30T06:26:53.107ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "72d0d647-2ce2-4472-aff7-03541c7d49ec", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139732025-04-30T06:26:53.108ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
139742025-04-30T06:26:53.113ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be64a2f8-726b-4a62-827e-3616fd9efa47", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139752025-04-30T06:26:53.114ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
139762025-04-30T06:26:53.117ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3cf86bd-5919-4c16-9d12-7d6c22061b02", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139772025-04-30T06:26:53.118ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
139782025-04-30T06:26:53.119ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05fdf1d9-a3b0-435a-a5b2-b845ed89b13f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139792025-04-30T06:26:53.120ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
139802025-04-30T06:26:53.120ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4923042-87e9-4c63-90e2-f4d5b9fbc03a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139812025-04-30T06:26:53.121ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
139822025-04-30T06:26:53.123ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "197fdab5-0412-4b48-b53b-3b100db00b2d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139832025-04-30T06:26:53.124ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
139842025-04-30T06:26:53.137ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
139852025-04-30T06:26:53.137ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7fb0f48-5a11-4140-847a-cabb6dc4e135", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139862025-04-30T06:26:53.138ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
139872025-04-30T06:26:53.142ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0386d0a-f568-4be2-93d4-200a14310b96", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139882025-04-30T06:26:53.142ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
139892025-04-30T06:26:53.147ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99d5290e-8671-47d9-8500-91e359c86f8b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139902025-04-30T06:26:53.148ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
139912025-04-30T06:26:53.150ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdd2829c-e30e-4602-aa9b-afb3810edf74", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139922025-04-30T06:26:53.150ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
139932025-04-30T06:26:53.156ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c083295d-7f71-47a5-8d7b-721085f9dd12", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139942025-04-30T06:26:53.156ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
139952025-04-30T06:26:53.158ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d731a06f-4fb5-490c-adb0-67d7830e13f7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139962025-04-30T06:26:53.158ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
139972025-04-30T06:26:53.160ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8466b62-3fe4-4975-bb9e-ce61e78233df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
139982025-04-30T06:26:53.160ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
139992025-04-30T06:26:53.161ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e803728-4b12-473f-854c-3795db2fe414", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140002025-04-30T06:26:53.162ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
140012025-04-30T06:26:53.163ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91ea4877-8bdc-421f-a7c7-8772468acd35", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140022025-04-30T06:26:53.163ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
140032025-04-30T06:26:53.164ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70597322-bafb-42fa-b0eb-5bb407cc7d34", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140042025-04-30T06:26:53.165ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
140052025-04-30T06:26:53.166ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfe17b42-66d3-44c3-b746-f919e4f9f868", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140062025-04-30T06:26:53.166ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
140072025-04-30T06:26:53.169ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3ffd99c-48ea-4bb3-82b1-cf66416b7107", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140082025-04-30T06:26:53.169ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
140092025-04-30T06:26:53.172ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c56c88e9-d17e-4456-b133-d8a329f37eb1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140102025-04-30T06:26:53.173ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
140112025-04-30T06:26:53.174ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3013fbb2-19ed-4ce8-a169-38e1816375b9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140122025-04-30T06:26:53.175ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
140132025-04-30T06:26:53.177ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "300fd2b5-04a6-416a-8f50-e418c3ed05c5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140142025-04-30T06:26:53.177ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
140152025-04-30T06:26:53.183ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "af5e17a9-db29-4033-9f3c-14b6ecea3cca", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140162025-04-30T06:26:53.184ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
140172025-04-30T06:26:53.185ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0ea6f17-ee8c-4c60-a7f2-16ee95d6dc0f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140182025-04-30T06:26:53.185ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
140192025-04-30T06:26:53.186ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e4da98e-3050-45c1-89ce-31025b4a5f45", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140202025-04-30T06:26:53.187ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
140212025-04-30T06:26:53.187ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
140222025-04-30T06:26:53.188ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140232025-04-30T06:26:53.188ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140242025-04-30T06:26:53.188ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
140252025-04-30T06:26:53.189ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140262025-04-30T06:26:53.189ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f90c433-d178-4582-ba5d-49b715df313f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140272025-04-30T06:26:53.189ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
140282025-04-30T06:26:53.191ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "396f6e62-158c-4848-832a-1844991a8082", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140292025-04-30T06:26:53.192ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
140302025-04-30T06:26:53.192ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140312025-04-30T06:26:53.202ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
140322025-04-30T06:26:53.202ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140332025-04-30T06:26:53.202ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
140342025-04-30T06:26:53.203ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140352025-04-30T06:26:53.204ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "354e7d5c-a6b8-42aa-912c-f2fed02d5f4d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140362025-04-30T06:26:53.205ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
140372025-04-30T06:26:53.205ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140382025-04-30T06:26:53.205ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140392025-04-30T06:26:53.205ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140402025-04-30T06:26:53.206ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140412025-04-30T06:26:53.206ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140422025-04-30T06:26:53.206ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
140432025-04-30T06:26:53.207ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140442025-04-30T06:26:53.208ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140452025-04-30T06:26:53.212ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
140462025-04-30T06:26:53.213ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
140472025-04-30T06:26:53.213ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140482025-04-30T06:26:53.213ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140492025-04-30T06:26:53.213ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140502025-04-30T06:26:53.213ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140512025-04-30T06:26:53.215ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140522025-04-30T06:26:53.220ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
140532025-04-30T06:26:53.220ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140542025-04-30T06:26:53.220ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140552025-04-30T06:26:53.220ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140562025-04-30T06:26:53.283ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
140572025-04-30T06:26:53.284ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
140582025-04-30T06:26:53.285ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:44515 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
140592025-04-30T06:26:53.286ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
140602025-04-30T06:26:53.286ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:44515 req_id = bd7fbc53-e6db-448c-8144-835bdc99efba sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
140612025-04-30T06:26:53.286ZINFOtest_disk_expunge (dropshot): request completed latency_us = 482 local_addr = [::1]:44559 method = GET remote_addr = [::1]:44515 req_id = bd7fbc53-e6db-448c-8144-835bdc99efba response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
140622025-04-30T06:26:53.288ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 84 63 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .c..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:49505 req_id = 3bd82602-2013-48a0-ad3f-09ab39a2d3fd
140632025-04-30T06:26:53.288ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33891,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49505 req_id = 3bd82602-2013-48a0-ad3f-09ab39a2d3fd
140642025-04-30T06:26:53.288ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
140652025-04-30T06:26:53.288ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
140662025-04-30T06:26:53.288ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49505 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3bd82602-2013-48a0-ad3f-09ab39a2d3fd
140672025-04-30T06:26:53.288ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd7fbc53-e6db-448c-8144-835bdc99efba", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
140682025-04-30T06:26:53.290ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
140692025-04-30T06:26:53.290ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
140702025-04-30T06:26:53.290ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
140712025-04-30T06:26:53.292ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
140722025-04-30T06:26:53.292ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140732025-04-30T06:26:53.292ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140742025-04-30T06:26:53.292ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140752025-04-30T06:26:53.293ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140762025-04-30T06:26:53.293ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3278, tv_nsec: 92515746 } })
140772025-04-30T06:26:53.293ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
140782025-04-30T06:26:53.294ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ae f2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63460 req_id = 759f52bc-02d5-4e0f-b4e4-c4d7c0595756
140792025-04-30T06:26:53.294ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44786,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63460 req_id = 759f52bc-02d5-4e0f-b4e4-c4d7c0595756
140802025-04-30T06:26:53.294ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
140812025-04-30T06:26:53.294ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63460 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 759f52bc-02d5-4e0f-b4e4-c4d7c0595756
140822025-04-30T06:26:53.295ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
140832025-04-30T06:26:53.370ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
140842025-04-30T06:26:53.370ZDEBGtest_disk_expunge (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
140852025-04-30T06:26:53.370ZDEBGtest_disk_expunge (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
140862025-04-30T06:26:53.370ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
140872025-04-30T06:26:53.373ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
140882025-04-30T06:26:53.376ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140892025-04-30T06:26:53.380ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
140902025-04-30T06:26:53.381ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140912025-04-30T06:26:53.381ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140922025-04-30T06:26:53.382ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140932025-04-30T06:26:53.405ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
140942025-04-30T06:26:53.412ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
140952025-04-30T06:26:53.413ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
140962025-04-30T06:26:53.413ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 563.401731ms iteration = 51 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
140972025-04-30T06:26:53.413ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 52 reason = Signaled
140982025-04-30T06:26:53.413ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
140992025-04-30T06:26:53.413ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141002025-04-30T06:26:53.413ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141012025-04-30T06:26:53.413ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141022025-04-30T06:26:53.414ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141032025-04-30T06:26:53.418ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141042025-04-30T06:26:53.426ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
141052025-04-30T06:26:53.426ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141062025-04-30T06:26:53.426ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141072025-04-30T06:26:53.427ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141082025-04-30T06:26:53.431ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
141092025-04-30T06:26:53.431ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
141102025-04-30T06:26:53.432ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 61 05 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 a............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65147 req_id = 1febad3e-7ad2-4b2d-baa8-7cf4e2453faa
141112025-04-30T06:26:53.432ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24837,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65147 req_id = 1febad3e-7ad2-4b2d-baa8-7cf4e2453faa
141122025-04-30T06:26:53.432ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
141132025-04-30T06:26:53.432ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
141142025-04-30T06:26:53.432ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65147 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1febad3e-7ad2-4b2d-baa8-7cf4e2453faa
141152025-04-30T06:26:53.433ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3278, tv_nsec: 232107944 } })
141162025-04-30T06:26:53.433ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
141172025-04-30T06:26:53.433ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 81 df 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49553 req_id = 01f9741a-0c38-4c4c-9142-97aff1ba0a88
141182025-04-30T06:26:53.433ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33247,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49553 req_id = 01f9741a-0c38-4c4c-9142-97aff1ba0a88
141192025-04-30T06:26:53.433ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
141202025-04-30T06:26:53.433ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49553 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 01f9741a-0c38-4c4c-9142-97aff1ba0a88
141212025-04-30T06:26:53.484ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
141222025-04-30T06:26:53.485ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
141232025-04-30T06:26:53.485ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141242025-04-30T06:26:53.485ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141252025-04-30T06:26:53.485ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
141262025-04-30T06:26:53.487ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141272025-04-30T06:26:53.487ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
141282025-04-30T06:26:53.487ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 05 48 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .H..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59693 req_id = 895e8f2f-f0e7-4808-a629-ef4a71314f07
141292025-04-30T06:26:53.487ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1352,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59693 req_id = 895e8f2f-f0e7-4808-a629-ef4a71314f07
141302025-04-30T06:26:53.487ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
141312025-04-30T06:26:53.487ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1352, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59693 req_id = 895e8f2f-f0e7-4808-a629-ef4a71314f07
141322025-04-30T06:26:53.488ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
141332025-04-30T06:26:53.488ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
141342025-04-30T06:26:53.488ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
141352025-04-30T06:26:53.488ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 0f d7 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:65455 req_id = 74a24a8b-1a2d-4621-b762-4f404942c3c9
141362025-04-30T06:26:53.488ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4055,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65455 req_id = 74a24a8b-1a2d-4621-b762-4f404942c3c9
141372025-04-30T06:26:53.488ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
141382025-04-30T06:26:53.488ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
141392025-04-30T06:26:53.488ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65455 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 74a24a8b-1a2d-4621-b762-4f404942c3c9
141402025-04-30T06:26:53.490ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3278, tv_nsec: 289030198 } })
141412025-04-30T06:26:53.490ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
141422025-04-30T06:26:53.491ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141432025-04-30T06:26:53.503ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
141442025-04-30T06:26:53.503ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141452025-04-30T06:26:53.503ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
141462025-04-30T06:26:53.504ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141472025-04-30T06:26:53.504ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 53 d6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e S...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57760 req_id = 6d59bc61-697c-4dbb-b852-272c80c624e6
141482025-04-30T06:26:53.504ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21462,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57760 req_id = 6d59bc61-697c-4dbb-b852-272c80c624e6
141492025-04-30T06:26:53.504ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
141502025-04-30T06:26:53.504ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57760 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6d59bc61-697c-4dbb-b852-272c80c624e6
141512025-04-30T06:26:53.505ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
141522025-04-30T06:26:53.573ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
141532025-04-30T06:26:53.573ZDEBGtest_disk_expunge (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
141542025-04-30T06:26:53.575ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "57a6b8c4-1dd6-4d59-92dc-b485ebcd46f4", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141552025-04-30T06:26:53.577ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
141562025-04-30T06:26:53.577ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141572025-04-30T06:26:53.577ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
141582025-04-30T06:26:53.578ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141592025-04-30T06:26:53.580ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c729430a-e5f4-401a-a2fd-51261146f807", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141602025-04-30T06:26:53.580ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
141612025-04-30T06:26:53.584ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a26b694-55d0-4e01-b583-5abc110e3738", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141622025-04-30T06:26:53.584ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
141632025-04-30T06:26:53.586ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa1b9d30-c3f2-4181-9bdb-dca1f60affb0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141642025-04-30T06:26:53.587ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
141652025-04-30T06:26:53.591ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca401e4e-3a38-489e-bcee-3c77b4572c7d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141662025-04-30T06:26:53.591ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
141672025-04-30T06:26:53.594ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4b9a54f-4937-4e4f-b4f2-fda9bdb3ecfd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141682025-04-30T06:26:53.594ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
141692025-04-30T06:26:53.599ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69feff54-2515-41bc-8e30-b1d701aa3354", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141702025-04-30T06:26:53.599ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
141712025-04-30T06:26:53.602ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "713ac164-9dfa-4f2c-b0ee-3f7e97c5ebf8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141722025-04-30T06:26:53.602ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
141732025-04-30T06:26:53.603ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06123ae9-6c04-448e-8c33-8875e0096af9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141742025-04-30T06:26:53.603ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
141752025-04-30T06:26:53.605ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "182315c0-3b6b-4024-9e6f-5649f6e9353c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141762025-04-30T06:26:53.605ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
141772025-04-30T06:26:53.606ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdff5e97-96ec-45f6-a309-b012ed836798", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141782025-04-30T06:26:53.607ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
141792025-04-30T06:26:53.612ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
141802025-04-30T06:26:53.612ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "063addfd-31b4-4332-b8f0-83e014b675e2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141812025-04-30T06:26:53.613ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
141822025-04-30T06:26:53.616ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dd855a9-e5d5-467c-b0e6-0a2d7da28c31", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141832025-04-30T06:26:53.616ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
141842025-04-30T06:26:53.620ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "45e951fd-b2d6-4a08-bf9c-35bb74b82560", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141852025-04-30T06:26:53.620ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
141862025-04-30T06:26:53.623ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1783ed88-934f-4f90-837c-1b1a1a80cad2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141872025-04-30T06:26:53.623ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
141882025-04-30T06:26:53.626ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "004b6348-17ea-4832-97c5-eb4174bc91f4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141892025-04-30T06:26:53.626ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
141902025-04-30T06:26:53.628ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a755cfc6-8425-4c49-bb5f-da197f14798e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141912025-04-30T06:26:53.629ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
141922025-04-30T06:26:53.630ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17c2e23f-0db3-44ed-914d-71cb12140c93", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141932025-04-30T06:26:53.631ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
141942025-04-30T06:26:53.633ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "858f9934-3c2d-4b63-862e-9fffc6193f1c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141952025-04-30T06:26:53.633ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
141962025-04-30T06:26:53.634ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f1de625-edd7-497e-8204-5a787021236b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141972025-04-30T06:26:53.634ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
141982025-04-30T06:26:53.635ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "482ecd6c-457f-4d5d-be38-9006b5e28f84", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
141992025-04-30T06:26:53.635ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
142002025-04-30T06:26:53.637ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96aec503-fe29-4131-8cbc-c75af198cc59", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142012025-04-30T06:26:53.637ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
142022025-04-30T06:26:53.638ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "952c09d3-3885-47b5-b103-1e320510e6d4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142032025-04-30T06:26:53.638ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
142042025-04-30T06:26:53.640ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ede4914a-bd8a-49d1-bb19-8a26fb0b9514", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142052025-04-30T06:26:53.640ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
142062025-04-30T06:26:53.643ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "23cc39b2-fbc5-4a4d-b0c3-994113c23d20", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142072025-04-30T06:26:53.643ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
142082025-04-30T06:26:53.645ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1e79105-a2fa-47e4-ab67-5ce16a0a566d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142092025-04-30T06:26:53.646ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
142102025-04-30T06:26:53.649ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50f32ebc-c07d-4c27-b916-2ea5e43d5d71", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142112025-04-30T06:26:53.649ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
142122025-04-30T06:26:53.651ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a9f01d9-4d02-4cab-a93a-3cbdd1211127", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142132025-04-30T06:26:53.651ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
142142025-04-30T06:26:53.654ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "47504e07-0cf0-44e2-9bb6-994637854178", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142152025-04-30T06:26:53.654ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
142162025-04-30T06:26:53.658ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e545a859-3392-4ace-a16d-98d00abce506", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142172025-04-30T06:26:53.659ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
142182025-04-30T06:26:53.661ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c23fb56-0b6c-4459-906d-5cb789e86030", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142192025-04-30T06:26:53.661ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
142202025-04-30T06:26:53.662ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30963434-2102-4886-acb6-53834c1d4af2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142212025-04-30T06:26:53.662ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
142222025-04-30T06:26:53.663ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
142232025-04-30T06:26:53.663ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
142242025-04-30T06:26:53.663ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142252025-04-30T06:26:53.663ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
142262025-04-30T06:26:53.666ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142272025-04-30T06:26:53.667ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f354b340-5d5a-4815-b051-89b7cdb09199", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142282025-04-30T06:26:53.667ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
142292025-04-30T06:26:53.668ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d086144-dcc4-4d80-9e23-6e5d338685b4", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142302025-04-30T06:26:53.668ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
142312025-04-30T06:26:53.669ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
142322025-04-30T06:26:53.675ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
142332025-04-30T06:26:53.675ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142342025-04-30T06:26:53.675ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
142352025-04-30T06:26:53.676ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142362025-04-30T06:26:53.679ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "613dd123-01e3-414c-b41b-b154851bbfe9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142372025-04-30T06:26:53.679ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
142382025-04-30T06:26:53.679ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142392025-04-30T06:26:53.680ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
142402025-04-30T06:26:53.681ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142412025-04-30T06:26:53.683ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:45341 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
142422025-04-30T06:26:53.683ZTRACtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
142432025-04-30T06:26:53.684ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:61503 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
142442025-04-30T06:26:53.684ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:45341 req_id = ac3fb1ca-c53e-4223-a58d-5c74f3c24d90 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142452025-04-30T06:26:53.685ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 446 local_addr = [::1]:47737 method = GET remote_addr = [::1]:45341 req_id = ac3fb1ca-c53e-4223-a58d-5c74f3c24d90 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142462025-04-30T06:26:53.685ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:61503 req_id = 0eeca3c0-c6d4-446e-8a11-5d6b3b0b592b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142472025-04-30T06:26:53.686ZTRACtest_disk_expunge (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 4e 8c 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 N............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:57288 req_id = 0c66c9c8-d45c-4887-9be3-3b3e5d08505e
142482025-04-30T06:26:53.686ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20108,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57288 req_id = 0c66c9c8-d45c-4887-9be3-3b3e5d08505e
142492025-04-30T06:26:53.687ZDEBGtest_disk_expunge (store): query key key = _dendrite._tcp
142502025-04-30T06:26:53.688ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
142512025-04-30T06:26:53.688ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57288 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0c66c9c8-d45c-4887-9be3-3b3e5d08505e
142522025-04-30T06:26:53.688ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 3244 local_addr = [::1]:47737 method = GET remote_addr = [::1]:61503 req_id = 0eeca3c0-c6d4-446e-8a11-5d6b3b0b592b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142532025-04-30T06:26:53.689ZDEBGtest_disk_expunge (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 55614, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3278, tv_nsec: 488374158 } })
142542025-04-30T06:26:53.689ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
142552025-04-30T06:26:53.691ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:46246 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
142562025-04-30T06:26:53.691ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:63161 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
142572025-04-30T06:26:53.691ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:53887 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
142582025-04-30T06:26:53.692ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:46246 req_id = 81d61c3c-318e-4451-9f99-52429e90d037 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142592025-04-30T06:26:53.692ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:63161 req_id = befb3cfe-944b-4ab5-a12c-8656dd5c5eac sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142602025-04-30T06:26:53.692ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:53887 req_id = 8216c696-d20b-4a7f-bf73-8ac5c44e7965 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142612025-04-30T06:26:53.692ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 3b 4a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ;J..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54143 req_id = 18391e18-b842-4d08-a4a7-133fbfab9ff7
142622025-04-30T06:26:53.693ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15178,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54143 req_id = 18391e18-b842-4d08-a4a7-133fbfab9ff7
142632025-04-30T06:26:53.693ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
142642025-04-30T06:26:53.693ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54143 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 18391e18-b842-4d08-a4a7-133fbfab9ff7
142652025-04-30T06:26:53.693ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 1337 local_addr = [::1]:47737 method = GET remote_addr = [::1]:46246 req_id = 81d61c3c-318e-4451-9f99-52429e90d037 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142662025-04-30T06:26:53.693ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 1092 local_addr = [::1]:47737 method = GET remote_addr = [::1]:63161 req_id = befb3cfe-944b-4ab5-a12c-8656dd5c5eac response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142672025-04-30T06:26:53.693ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 1004 local_addr = [::1]:47737 method = GET remote_addr = [::1]:53887 req_id = 8216c696-d20b-4a7f-bf73-8ac5c44e7965 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142682025-04-30T06:26:53.694ZINFOtest_disk_expunge (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
142692025-04-30T06:26:53.767ZINFOtest_disk_expunge (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
142702025-04-30T06:26:53.768ZDEBGtest_disk_expunge (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
142712025-04-30T06:26:53.769ZDEBGtest_disk_expunge (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
142722025-04-30T06:26:53.769ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
142732025-04-30T06:26:53.770ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
142742025-04-30T06:26:53.774ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:46713 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
142752025-04-30T06:26:53.782ZINFOtest_disk_expunge (dropshot): accepted connection kind = pantry local_addr = [::1]:47737 remote_addr = [::1]:39492 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
142762025-04-30T06:26:53.782ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:46713 req_id = 7cf4e4c9-7e54-43d3-b081-cc5a418b4cbd sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142772025-04-30T06:26:53.783ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 685 local_addr = [::1]:47737 method = GET remote_addr = [::1]:46713 req_id = 7cf4e4c9-7e54-43d3-b081-cc5a418b4cbd response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142782025-04-30T06:26:53.785ZTRACtest_disk_expunge (dropshot): incoming request kind = pantry local_addr = [::1]:47737 method = GET remote_addr = [::1]:39492 req_id = c7993190-a036-4d80-9506-128c37420584 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142792025-04-30T06:26:53.786ZINFOtest_disk_expunge (dropshot): request completed kind = pantry latency_us = 1050 local_addr = [::1]:47737 method = GET remote_addr = [::1]:39492 req_id = c7993190-a036-4d80-9506-128c37420584 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
142802025-04-30T06:26:53.786ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22488320-578d-414a-bb86-26c115174b5c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142812025-04-30T06:26:53.786ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
142822025-04-30T06:26:53.797ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5648e901-9f0d-495d-bf85-c4319ed7ab47", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142832025-04-30T06:26:53.798ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
142842025-04-30T06:26:53.802ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebeb1672-c8c5-4e00-b43d-239057946587", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142852025-04-30T06:26:53.803ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
142862025-04-30T06:26:53.807ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00d01703-5b40-4934-8945-310fd70c2731", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142872025-04-30T06:26:53.807ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
142882025-04-30T06:26:53.813ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
142892025-04-30T06:26:53.816ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "821f1da6-699a-4b13-bac3-cadc1387f1ae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142902025-04-30T06:26:53.816ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
142912025-04-30T06:26:53.818ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2615826e-8896-4f02-993a-ab9fcfd02f16", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142922025-04-30T06:26:53.819ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
142932025-04-30T06:26:53.821ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e1dfc26-78b7-4ab2-b9c3-4cd8a7fccfe4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142942025-04-30T06:26:53.822ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
142952025-04-30T06:26:53.823ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "770f31c3-7381-4ffb-8ab6-a4d4e9253476", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142962025-04-30T06:26:53.823ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
142972025-04-30T06:26:53.828ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16dd309a-f448-48d2-9b65-4d913e626450", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
142982025-04-30T06:26:53.829ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
142992025-04-30T06:26:53.832ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c9fe116-e5d7-40c4-bc5d-a56f021a178b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:52 GMT"} })
143002025-04-30T06:26:53.832ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
143012025-04-30T06:26:53.832ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143022025-04-30T06:26:53.832ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143032025-04-30T06:26:53.832ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143042025-04-30T06:26:53.833ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143052025-04-30T06:26:53.838ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143062025-04-30T06:26:53.847ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
143072025-04-30T06:26:53.847ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
143082025-04-30T06:26:53.847ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143092025-04-30T06:26:53.849ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143102025-04-30T06:26:53.849ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143112025-04-30T06:26:53.850ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143122025-04-30T06:26:53.854ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143132025-04-30T06:26:53.865ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
143142025-04-30T06:26:53.865ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143152025-04-30T06:26:53.865ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143162025-04-30T06:26:53.866ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143172025-04-30T06:26:53.867ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
143182025-04-30T06:26:53.867ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143192025-04-30T06:26:53.867ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143202025-04-30T06:26:53.867ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
143212025-04-30T06:26:53.868ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143222025-04-30T06:26:53.872ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143232025-04-30T06:26:53.878ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
143242025-04-30T06:26:53.878ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143252025-04-30T06:26:53.878ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
143262025-04-30T06:26:53.879ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143272025-04-30T06:26:53.953ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
143282025-04-30T06:26:53.953ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
143292025-04-30T06:26:53.954ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:57002 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143302025-04-30T06:26:53.955ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:57002 req_id = 90a075f4-e4bd-4771-9e69-8b88c6c14050 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
143312025-04-30T06:26:53.956ZINFOtest_disk_expunge (dropshot): request completed latency_us = 1412 local_addr = [::1]:44559 method = GET remote_addr = [::1]:57002 req_id = 90a075f4-e4bd-4771-9e69-8b88c6c14050 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
143322025-04-30T06:26:53.957ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143332025-04-30T06:26:53.957ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
143342025-04-30T06:26:53.957ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143352025-04-30T06:26:53.957ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
143362025-04-30T06:26:53.958ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
143372025-04-30T06:26:53.958ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "90a075f4-e4bd-4771-9e69-8b88c6c14050", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
143382025-04-30T06:26:53.960ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
143392025-04-30T06:26:53.960ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
143402025-04-30T06:26:53.960ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
143412025-04-30T06:26:53.962ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
143422025-04-30T06:26:53.962ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143432025-04-30T06:26:53.962ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143442025-04-30T06:26:53.962ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143452025-04-30T06:26:53.963ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143462025-04-30T06:26:53.963ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
143472025-04-30T06:26:53.964ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
143482025-04-30T06:26:53.969ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
143492025-04-30T06:26:53.969ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
143502025-04-30T06:26:53.969ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143512025-04-30T06:26:53.974ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
143522025-04-30T06:26:53.974ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143532025-04-30T06:26:53.974ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143542025-04-30T06:26:53.975ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143552025-04-30T06:26:53.978ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
143562025-04-30T06:26:53.978ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
143572025-04-30T06:26:54.020ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
143582025-04-30T06:26:54.022ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
143592025-04-30T06:26:54.024ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
143602025-04-30T06:26:54.024ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 611.575234ms iteration = 52 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
143612025-04-30T06:26:54.024ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 53 reason = Signaled
143622025-04-30T06:26:54.024ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
143632025-04-30T06:26:54.024ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143642025-04-30T06:26:54.025ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143652025-04-30T06:26:54.025ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143662025-04-30T06:26:54.028ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143672025-04-30T06:26:54.045ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143682025-04-30T06:26:54.053ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
143692025-04-30T06:26:54.053ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143702025-04-30T06:26:54.053ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143712025-04-30T06:26:54.054ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143722025-04-30T06:26:54.056ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143732025-04-30T06:26:54.056ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143742025-04-30T06:26:54.066ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
143752025-04-30T06:26:54.066ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
143762025-04-30T06:26:54.067ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143772025-04-30T06:26:54.068ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143782025-04-30T06:26:54.068ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143792025-04-30T06:26:54.068ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
143802025-04-30T06:26:54.068ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
143812025-04-30T06:26:54.072ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 24 fc 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 $............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61263 req_id = 6df4eed3-7960-40ca-a20d-15838cb2e060
143822025-04-30T06:26:54.072ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9468,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61263 req_id = 6df4eed3-7960-40ca-a20d-15838cb2e060
143832025-04-30T06:26:54.072ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
143842025-04-30T06:26:54.072ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
143852025-04-30T06:26:54.073ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61263 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6df4eed3-7960-40ca-a20d-15838cb2e060
143862025-04-30T06:26:54.074ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
143872025-04-30T06:26:54.074ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143882025-04-30T06:26:54.074ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143892025-04-30T06:26:54.074ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
143902025-04-30T06:26:54.085ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143912025-04-30T06:26:54.086ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3278, tv_nsec: 885064577 } })
143922025-04-30T06:26:54.086ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
143932025-04-30T06:26:54.090ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7e 46 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ~F..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56525 req_id = 50105855-117f-4666-aeb0-70fdcf149bed
143942025-04-30T06:26:54.090ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32326,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56525 req_id = 50105855-117f-4666-aeb0-70fdcf149bed
143952025-04-30T06:26:54.090ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
143962025-04-30T06:26:54.090ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56525 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 50105855-117f-4666-aeb0-70fdcf149bed
143972025-04-30T06:26:54.161ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
143982025-04-30T06:26:54.166ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143992025-04-30T06:26:54.172ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
144002025-04-30T06:26:54.172ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144012025-04-30T06:26:54.172ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
144022025-04-30T06:26:54.173ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144032025-04-30T06:26:54.177ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 70 b9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c p............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54552 req_id = 60f4a275-301e-4907-a51f-86a0509d28f0
144042025-04-30T06:26:54.178ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28857,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54552 req_id = 60f4a275-301e-4907-a51f-86a0509d28f0
144052025-04-30T06:26:54.179ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
144062025-04-30T06:26:54.188ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28857, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54552 req_id = 60f4a275-301e-4907-a51f-86a0509d28f0
144072025-04-30T06:26:54.189ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
144082025-04-30T06:26:54.189ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
144092025-04-30T06:26:54.189ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
144102025-04-30T06:26:54.195ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144112025-04-30T06:26:54.195ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
144122025-04-30T06:26:54.196ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144132025-04-30T06:26:54.203ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e148b24-8506-4ad9-b371-5b04c9d68462", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144142025-04-30T06:26:54.203ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
144152025-04-30T06:26:54.212ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d0fd4c3-a6c6-4124-982d-2802f39bcbc0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144162025-04-30T06:26:54.212ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
144172025-04-30T06:26:54.217ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b68450d9-e491-4e3a-9625-42ed1968725a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144182025-04-30T06:26:54.217ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
144192025-04-30T06:26:54.220ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c58bcafa-edd7-4dd4-a9c4-7ebcdc03bda1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144202025-04-30T06:26:54.220ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
144212025-04-30T06:26:54.222ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "35feddb6-9599-4019-9708-e16a111c7757", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144222025-04-30T06:26:54.222ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
144232025-04-30T06:26:54.225ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2c3a22d-7023-4eb4-85d4-6bd42d8afb86", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144242025-04-30T06:26:54.225ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
144252025-04-30T06:26:54.227ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15051fc4-269c-4538-9cef-50d3eb19a5cc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144262025-04-30T06:26:54.227ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
144272025-04-30T06:26:54.234ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
144282025-04-30T06:26:54.236ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b6075c0-b11b-40c3-a341-e2847718f8d0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144292025-04-30T06:26:54.236ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
144302025-04-30T06:26:54.237ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ebcd5a1-c9aa-48d1-90fc-4ee4c3f23200", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144312025-04-30T06:26:54.237ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
144322025-04-30T06:26:54.239ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a35734df-f127-4b88-95f7-a77ad8c0657f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144332025-04-30T06:26:54.239ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
144342025-04-30T06:26:54.240ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1106d2b2-027f-4700-80e9-38e7e1523b6b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144352025-04-30T06:26:54.240ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
144362025-04-30T06:26:54.242ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6fe3b3e-b2ad-4f49-9e5c-94ac2dce339b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144372025-04-30T06:26:54.242ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
144382025-04-30T06:26:54.244ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bc2ed4e-4080-4c01-b8ef-fd49f868b2c1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144392025-04-30T06:26:54.244ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
144402025-04-30T06:26:54.247ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f01e1f2-f448-412d-b33a-f8f2769756eb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144412025-04-30T06:26:54.247ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
144422025-04-30T06:26:54.251ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fee585f-cf26-4810-80d4-ec39371e4999", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144432025-04-30T06:26:54.251ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
144442025-04-30T06:26:54.254ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4adf2b5d-1bf4-4b2c-8d91-0be5c39ab2b7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144452025-04-30T06:26:54.254ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
144462025-04-30T06:26:54.256ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e01750b7-3c6e-48ea-9de8-2a29309ee9b1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144472025-04-30T06:26:54.256ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
144482025-04-30T06:26:54.258ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6847d2d-87a1-4dc2-873e-02d8f0ab3296", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144492025-04-30T06:26:54.258ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
144502025-04-30T06:26:54.260ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9b7ddee-a87d-41bd-8e0d-d19b26fd3323", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144512025-04-30T06:26:54.260ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
144522025-04-30T06:26:54.261ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1cd0b1a-e995-42f0-9ccc-8496c604c963", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144532025-04-30T06:26:54.261ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
144542025-04-30T06:26:54.262ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "13f4fd8d-18f6-4702-9d0f-92b4fef1b01e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144552025-04-30T06:26:54.262ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
144562025-04-30T06:26:54.263ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38143158-c79f-40e3-a682-8616db91f013", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144572025-04-30T06:26:54.263ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
144582025-04-30T06:26:54.264ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01a2ce11-ca3c-4a1d-a077-9539b372d770", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144592025-04-30T06:26:54.264ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
144602025-04-30T06:26:54.267ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d55d80b-2a5c-4a74-b7b0-4709ed85fa79", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144612025-04-30T06:26:54.268ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
144622025-04-30T06:26:54.270ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b42187f5-583b-4c64-b9f0-9b9a0d040042", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144632025-04-30T06:26:54.270ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
144642025-04-30T06:26:54.271ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d73d5afa-830f-41df-94b7-4d317c705020", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144652025-04-30T06:26:54.272ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
144662025-04-30T06:26:54.274ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29de209d-6bf9-407f-b3ab-b3a27770d241", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144672025-04-30T06:26:54.274ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
144682025-04-30T06:26:54.278ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f26b42d-1ed3-46a5-948c-78412e95ed38", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144692025-04-30T06:26:54.278ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
144702025-04-30T06:26:54.280ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac2ac4b3-1c8c-44f2-ac7c-759eea2ca54e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144712025-04-30T06:26:54.280ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
144722025-04-30T06:26:54.282ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77764a80-ae2e-4967-93e3-4ec1d0d79750", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144732025-04-30T06:26:54.282ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
144742025-04-30T06:26:54.284ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3125803d-c9bf-473e-97d6-2f65fcb5d857", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144752025-04-30T06:26:54.284ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
144762025-04-30T06:26:54.286ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
144772025-04-30T06:26:54.286ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
144782025-04-30T06:26:54.286ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144792025-04-30T06:26:54.286ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
144802025-04-30T06:26:54.287ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144812025-04-30T06:26:54.288ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c287a0c2-95cd-4c79-a9a6-502de0be0885", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144822025-04-30T06:26:54.288ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
144832025-04-30T06:26:54.291ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
144842025-04-30T06:26:54.299ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
144852025-04-30T06:26:54.299ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144862025-04-30T06:26:54.299ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
144872025-04-30T06:26:54.299ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144882025-04-30T06:26:54.300ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b458d095-ae44-4e11-838a-cdbbfa14ce31", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144892025-04-30T06:26:54.300ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
144902025-04-30T06:26:54.303ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
144912025-04-30T06:26:54.303ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
144922025-04-30T06:26:54.306ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
144932025-04-30T06:26:54.307ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fe51cbf-11fa-4529-a0d2-ca674d5c5c35", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144942025-04-30T06:26:54.307ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
144952025-04-30T06:26:54.309ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd494973-ba82-4468-b99e-bdfa4baf0954", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144962025-04-30T06:26:54.310ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
144972025-04-30T06:26:54.317ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5369b3d0-c3ea-44f2-b020-b8e9cb83d093", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
144982025-04-30T06:26:54.318ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
144992025-04-30T06:26:54.325ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "732a1139-ac32-4fa1-9503-2ed329b3a9d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
145002025-04-30T06:26:54.325ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
145012025-04-30T06:26:54.331ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "130505fb-32db-4783-a846-e67675ed0fae", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
145022025-04-30T06:26:54.331ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
145032025-04-30T06:26:54.335ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2279a4fe-fb89-4f75-87ba-422c004b66f9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
145042025-04-30T06:26:54.336ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
145052025-04-30T06:26:54.343ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92a71c17-90d8-41d5-bfc8-559b911f9aa8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
145062025-04-30T06:26:54.343ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
145072025-04-30T06:26:54.348ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20fdb778-783d-4896-a649-c5c54e642343", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
145082025-04-30T06:26:54.348ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
145092025-04-30T06:26:54.352ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
145102025-04-30T06:26:54.352ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f949b33-9739-4cba-8e74-6c20c95735ee", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
145112025-04-30T06:26:54.353ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
145122025-04-30T06:26:54.354ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e830603-dd59-4e4c-9f55-35e5329d4690", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
145132025-04-30T06:26:54.355ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
145142025-04-30T06:26:54.366ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb47aa04-3b27-4413-a15d-21e8328ef2fc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
145152025-04-30T06:26:54.366ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
145162025-04-30T06:26:54.368ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6805a8e-86e4-4c65-a83a-528b7c042322", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
145172025-04-30T06:26:54.368ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
145182025-04-30T06:26:54.368ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145192025-04-30T06:26:54.368ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145202025-04-30T06:26:54.368ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145212025-04-30T06:26:54.369ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145222025-04-30T06:26:54.375ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145232025-04-30T06:26:54.388ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
145242025-04-30T06:26:54.388ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
145252025-04-30T06:26:54.388ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145262025-04-30T06:26:54.388ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145272025-04-30T06:26:54.389ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145282025-04-30T06:26:54.395ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145292025-04-30T06:26:54.397ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145302025-04-30T06:26:54.408ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
145312025-04-30T06:26:54.408ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145322025-04-30T06:26:54.408ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145332025-04-30T06:26:54.408ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145342025-04-30T06:26:54.409ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
145352025-04-30T06:26:54.409ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145362025-04-30T06:26:54.409ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145372025-04-30T06:26:54.409ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
145382025-04-30T06:26:54.410ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145392025-04-30T06:26:54.539ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
145402025-04-30T06:26:54.539ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
145412025-04-30T06:26:54.540ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:52698 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
145422025-04-30T06:26:54.541ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:52698 req_id = b1e1d2eb-615a-45c5-af26-69f90d23cbdb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
145432025-04-30T06:26:54.542ZINFOtest_disk_expunge (dropshot): request completed latency_us = 783 local_addr = [::1]:44559 method = GET remote_addr = [::1]:52698 req_id = b1e1d2eb-615a-45c5-af26-69f90d23cbdb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
145442025-04-30T06:26:54.543ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1e1d2eb-615a-45c5-af26-69f90d23cbdb", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
145452025-04-30T06:26:54.546ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
145462025-04-30T06:26:54.546ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
145472025-04-30T06:26:54.546ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
145482025-04-30T06:26:54.549ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
145492025-04-30T06:26:54.549ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145502025-04-30T06:26:54.549ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145512025-04-30T06:26:54.549ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145522025-04-30T06:26:54.550ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145532025-04-30T06:26:54.550ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145542025-04-30T06:26:54.556ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
145552025-04-30T06:26:54.556ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145562025-04-30T06:26:54.556ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
145572025-04-30T06:26:54.557ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145582025-04-30T06:26:54.558ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145592025-04-30T06:26:54.562ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
145602025-04-30T06:26:54.563ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145612025-04-30T06:26:54.563ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145622025-04-30T06:26:54.563ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145632025-04-30T06:26:54.564ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145642025-04-30T06:26:54.564ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
145652025-04-30T06:26:54.565ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
145662025-04-30T06:26:54.608ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
145672025-04-30T06:26:54.608ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
145682025-04-30T06:26:54.608ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 583.730506ms iteration = 53 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
145692025-04-30T06:26:54.608ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 54 reason = Signaled
145702025-04-30T06:26:54.608ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
145712025-04-30T06:26:54.608ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145722025-04-30T06:26:54.608ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145732025-04-30T06:26:54.608ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145742025-04-30T06:26:54.609ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145752025-04-30T06:26:54.613ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
145762025-04-30T06:26:54.616ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
145772025-04-30T06:26:54.624ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
145782025-04-30T06:26:54.624ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
145792025-04-30T06:26:54.624ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
145802025-04-30T06:26:54.625ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
145812025-04-30T06:26:54.628ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
145822025-04-30T06:26:54.629ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
145832025-04-30T06:26:54.630ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 33 ae 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 3............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56518 req_id = 319b3e7d-af26-42c7-bff7-570c1179f459
145842025-04-30T06:26:54.630ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13230,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56518 req_id = 319b3e7d-af26-42c7-bff7-570c1179f459
145852025-04-30T06:26:54.630ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
145862025-04-30T06:26:54.631ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
145872025-04-30T06:26:54.631ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56518 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 319b3e7d-af26-42c7-bff7-570c1179f459
145882025-04-30T06:26:54.632ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3279, tv_nsec: 430696032 } })
145892025-04-30T06:26:54.633ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
145902025-04-30T06:26:54.634ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0e 55 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .U..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55181 req_id = 2554f5a8-5c39-41bd-8225-4dd532f4aee0
145912025-04-30T06:26:54.634ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3669,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55181 req_id = 2554f5a8-5c39-41bd-8225-4dd532f4aee0
145922025-04-30T06:26:54.634ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
145932025-04-30T06:26:54.634ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55181 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2554f5a8-5c39-41bd-8225-4dd532f4aee0
145942025-04-30T06:26:54.687ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
145952025-04-30T06:26:54.687ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
145962025-04-30T06:26:54.687ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
145972025-04-30T06:26:54.687ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
145982025-04-30T06:26:54.687ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
145992025-04-30T06:26:54.688ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146002025-04-30T06:26:54.688ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ef 5b 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .[..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63649 req_id = 21a11dd3-b577-462f-a365-bd8c89610ee8
146012025-04-30T06:26:54.688ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61275,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63649 req_id = 21a11dd3-b577-462f-a365-bd8c89610ee8
146022025-04-30T06:26:54.688ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
146032025-04-30T06:26:54.690ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 61275, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63649 req_id = 21a11dd3-b577-462f-a365-bd8c89610ee8
146042025-04-30T06:26:54.690ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
146052025-04-30T06:26:54.690ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
146062025-04-30T06:26:54.690ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
146072025-04-30T06:26:54.691ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146082025-04-30T06:26:54.700ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
146092025-04-30T06:26:54.701ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146102025-04-30T06:26:54.701ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
146112025-04-30T06:26:54.702ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146122025-04-30T06:26:54.704ZDEBGtest_disk_expunge (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
146132025-04-30T06:26:54.704ZWARNtest_disk_expunge (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
146142025-04-30T06:26:54.708ZWARNtest_disk_expunge (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
146152025-04-30T06:26:54.709ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "82511d6f-54b0-4c3a-a018-bec4bb177114", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146162025-04-30T06:26:54.710ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
146172025-04-30T06:26:54.710ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146182025-04-30T06:26:54.710ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
146192025-04-30T06:26:54.711ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146202025-04-30T06:26:54.712ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "793339c2-72e5-4085-aa07-59c222f6be84", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146212025-04-30T06:26:54.712ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
146222025-04-30T06:26:54.715ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3ab7005-b395-4b05-8e72-f80d96f177f6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146232025-04-30T06:26:54.715ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
146242025-04-30T06:26:54.717ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b508830-8203-4dd1-bf04-114a8d1cc73f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146252025-04-30T06:26:54.717ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
146262025-04-30T06:26:54.719ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cb78295-939f-49e7-9c1c-83b130879a84", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146272025-04-30T06:26:54.720ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
146282025-04-30T06:26:54.723ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afc54385-17ab-4d97-9e6e-f51ada6b6f0f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146292025-04-30T06:26:54.723ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
146302025-04-30T06:26:54.725ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61996c3d-c34a-41c3-bb45-c6b5953283bc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146312025-04-30T06:26:54.725ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
146322025-04-30T06:26:54.728ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04177ae6-8f24-49ac-8b5c-d9a2ac48b4a9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146332025-04-30T06:26:54.728ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
146342025-04-30T06:26:54.732ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "332f4f4e-cdfa-41df-8e2f-dddc9b10fc36", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146352025-04-30T06:26:54.732ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
146362025-04-30T06:26:54.734ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5960b604-6f49-42a3-ab5d-ae2e3d864f61", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146372025-04-30T06:26:54.734ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
146382025-04-30T06:26:54.738ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f000244b-3eb5-44e2-9a2d-42bd83018f59", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146392025-04-30T06:26:54.738ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
146402025-04-30T06:26:54.741ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "514e77fc-358b-4baa-93c9-416dcf9baa71", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146412025-04-30T06:26:54.741ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
146422025-04-30T06:26:54.747ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
146432025-04-30T06:26:54.748ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c15ffac-0f12-4070-b9cc-8f890039f8bd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146442025-04-30T06:26:54.748ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
146452025-04-30T06:26:54.752ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b90a241d-913d-4b8e-a188-209e398237b6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146462025-04-30T06:26:54.752ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
146472025-04-30T06:26:54.756ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e62a4e39-beba-4930-9ee7-4f844dada9d8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146482025-04-30T06:26:54.757ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
146492025-04-30T06:26:54.760ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "24523ba0-eb80-4d76-929c-4abbc2efcf05", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146502025-04-30T06:26:54.760ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
146512025-04-30T06:26:54.766ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8be8079-cee6-4e37-8114-e015935b5136", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146522025-04-30T06:26:54.766ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0
146532025-04-30T06:26:54.768ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e9b1d27-238a-4e0d-85e6-8a3d7dd7e085", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146542025-04-30T06:26:54.768ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1
146552025-04-30T06:26:54.770ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f804315-c3ac-43ad-a6d3-767844bac9a4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146562025-04-30T06:26:54.770ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cmpa
146572025-04-30T06:26:54.771ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a9121c3-fd33-4050-bc73-6c376b8dbda8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146582025-04-30T06:26:54.771ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
146592025-04-30T06:26:54.773ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91392686-8fd0-4a3d-8467-2a09bd04e25e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146602025-04-30T06:26:54.773ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
146612025-04-30T06:26:54.775ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61548958-45f9-49c7-a1d6-26867087cb26", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146622025-04-30T06:26:54.775ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/cfpa
146632025-04-30T06:26:54.776ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaae01f7-1e94-4b52-be18-e023eda08037", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146642025-04-30T06:26:54.776ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0
146652025-04-30T06:26:54.778ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b739b485-1664-4f98-88a1-8d42549da4bc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146662025-04-30T06:26:54.778ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0
146672025-04-30T06:26:54.780ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca527620-002b-454c-8283-85fa2002154f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146682025-04-30T06:26:54.780ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1
146692025-04-30T06:26:54.782ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f51c3c4b-898c-4e1d-809e-610f1d4e2caa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146702025-04-30T06:26:54.782ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0
146712025-04-30T06:26:54.785ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06129bc6-c5f9-45e4-887d-c48e0c28a1f8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146722025-04-30T06:26:54.785ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1
146732025-04-30T06:26:54.789ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cedf0b54-1759-4991-ac20-1f07a5401b87", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146742025-04-30T06:26:54.789ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0
146752025-04-30T06:26:54.794ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01db97f6-182e-4683-bbf5-69345bfe8a5b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146762025-04-30T06:26:54.795ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1
146772025-04-30T06:26:54.798ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
146782025-04-30T06:26:54.798ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
146792025-04-30T06:26:54.798ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146802025-04-30T06:26:54.798ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
146812025-04-30T06:26:54.801ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146822025-04-30T06:26:54.805ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71cfaf21-2552-4e95-8b34-faed287f713e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146832025-04-30T06:26:54.807ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cmpa
146842025-04-30T06:26:54.807ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
146852025-04-30T06:26:54.819ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
146862025-04-30T06:26:54.819ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146872025-04-30T06:26:54.819ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
146882025-04-30T06:26:54.820ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146892025-04-30T06:26:54.823ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e94eca5-7e43-4a4d-9444-bffcd2ab8e3c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146902025-04-30T06:26:54.823ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
146912025-04-30T06:26:54.823ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
146922025-04-30T06:26:54.824ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
146932025-04-30T06:26:54.824ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
146942025-04-30T06:26:54.824ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f39fe1b7-ac21-41b8-8c40-ae41609879a4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146952025-04-30T06:26:54.824ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
146962025-04-30T06:26:54.827ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b722dce9-2463-4ebe-b9a1-97d997b92108", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146972025-04-30T06:26:54.827ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/0/component/rot/cfpa
146982025-04-30T06:26:54.828ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d9f33f2-8540-4e8c-826d-c8b5055cef78", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
146992025-04-30T06:26:54.828ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1
147002025-04-30T06:26:54.831ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08dd1b35-7e3d-47a0-9273-6e1f5e324274", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
147012025-04-30T06:26:54.831ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0
147022025-04-30T06:26:54.837ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f38bcf47-373e-43f7-b1f0-42fc2846ad2b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
147032025-04-30T06:26:54.837ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1
147042025-04-30T06:26:54.842ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "101b216c-0be9-4519-bb74-c1630c1bef66", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
147052025-04-30T06:26:54.842ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0
147062025-04-30T06:26:54.846ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a71abe55-d1ab-49dc-bd5c-7b09119b6bcc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
147072025-04-30T06:26:54.847ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1
147082025-04-30T06:26:54.849ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a004ecb9-0db2-4ae4-b9a6-c8af6ee3e473", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
147092025-04-30T06:26:54.849ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0
147102025-04-30T06:26:54.854ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
147112025-04-30T06:26:54.858ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a1b6eb0-f7cf-467d-80e1-e2575e1a0ddf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
147122025-04-30T06:26:54.858ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1
147132025-04-30T06:26:54.862ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8b17545-b089-4577-9e64-ba2826eef853", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
147142025-04-30T06:26:54.862ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cmpa
147152025-04-30T06:26:54.864ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3cb9519-1225-4588-a42c-453f22876c5b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
147162025-04-30T06:26:54.864ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
147172025-04-30T06:26:54.865ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9bf2889-97c5-45c0-8c06-8d5cbcd95d56", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
147182025-04-30T06:26:54.865ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
147192025-04-30T06:26:54.867ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "999aafc2-d124-4793-aa8c-bfe30dbadb70", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
147202025-04-30T06:26:54.867ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/sled/1/component/rot/cfpa
147212025-04-30T06:26:54.868ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "199aa7b1-79bc-4b8b-bc1a-54f66c90d602", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
147222025-04-30T06:26:54.868ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
147232025-04-30T06:26:54.868ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147242025-04-30T06:26:54.868ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
147252025-04-30T06:26:54.868ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
147262025-04-30T06:26:54.869ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
147272025-04-30T06:26:54.872ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
147282025-04-30T06:26:54.881ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
147292025-04-30T06:26:54.881ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
147302025-04-30T06:26:54.881ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147312025-04-30T06:26:54.881ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
147322025-04-30T06:26:54.881ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
147332025-04-30T06:26:54.882ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
147342025-04-30T06:26:54.886ZINFOtest_disk_expunge (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
147352025-04-30T06:26:54.887ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
147362025-04-30T06:26:54.887ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f6fc2d1c-e56d-4771-a994-1c5e3547233f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
147372025-04-30T06:26:54.889ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f6fc2d1c-e56d-4771-a994-1c5e3547233f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
147382025-04-30T06:26:54.889ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f6fc2d1c-e56d-4771-a994-1c5e3547233f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
147392025-04-30T06:26:54.890ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f6fc2d1c-e56d-4771-a994-1c5e3547233f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
147402025-04-30T06:26:54.892ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
147412025-04-30T06:26:54.901ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
147422025-04-30T06:26:54.901ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
147432025-04-30T06:26:54.901ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
147442025-04-30T06:26:54.901ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
147452025-04-30T06:26:54.902ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 14605 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = f6fc2d1c-e56d-4771-a994-1c5e3547233f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
147462025-04-30T06:26:54.902ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6fc2d1c-e56d-4771-a994-1c5e3547233f", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:53 GMT"} })
147472025-04-30T06:26:54.902ZDEBGtest_disk_expunge (oximeter): client request body = None method = GET uri = http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
147482025-04-30T06:26:54.904ZTRACtest_disk_expunge (dropshot_internal): incoming request local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = fa9d5e93-6582-4b3b-a66b-e7817ab5f69c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
147492025-04-30T06:26:54.904ZTRACtest_disk_expunge (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = fa9d5e93-6582-4b3b-a66b-e7817ab5f69c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
147502025-04-30T06:26:54.904ZDEBGtest_disk_expunge (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = fa9d5e93-6582-4b3b-a66b-e7817ab5f69c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
147512025-04-30T06:26:54.904ZDEBGtest_disk_expunge (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = fa9d5e93-6582-4b3b-a66b-e7817ab5f69c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
147522025-04-30T06:26:54.905ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
147532025-04-30T06:26:54.905ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147542025-04-30T06:26:54.906ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147552025-04-30T06:26:54.906ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
147562025-04-30T06:26:54.907ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147572025-04-30T06:26:54.989ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:44559
147582025-04-30T06:26:54.989ZDEBGtest_disk_expunge (BackgroundTasks): client request SledAgent = http://[::1]:44559 background_task = inventory_collection body = None method = GET uri = http://[::1]:44559/inventory
147592025-04-30T06:26:54.991ZINFOtest_disk_expunge (dropshot): accepted connection local_addr = [::1]:44559 remote_addr = [::1]:62034 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
147602025-04-30T06:26:54.992ZDEBGtest_disk_expunge (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
147612025-04-30T06:26:54.992ZDEBGtest_disk_expunge (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
147622025-04-30T06:26:54.993ZDEBGtest_disk_expunge (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8
147632025-04-30T06:26:54.994ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
147642025-04-30T06:26:54.994ZINFOtest_disk_expunge (dropshot_internal): request completed latency_us = 91359 local_addr = 127.0.0.1:34036 method = GET remote_addr = 127.0.0.1:58228 req_id = fa9d5e93-6582-4b3b-a66b-e7817ab5f69c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
147652025-04-30T06:26:54.995ZTRACtest_disk_expunge (dropshot): incoming request local_addr = [::1]:44559 method = GET remote_addr = [::1]:62034 req_id = acb789a9-351b-42c6-9590-ab9d88f432e5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
147662025-04-30T06:26:54.995ZDEBGtest_disk_expunge (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:34036/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa9d5e93-6582-4b3b-a66b-e7817ab5f69c", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
147672025-04-30T06:26:54.995ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 78121327-166d-4e33-935c-82ab8635bf33
147682025-04-30T06:26:54.995ZTRACtest_disk_expunge (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
147692025-04-30T06:26:54.995ZINFOtest_disk_expunge (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
147702025-04-30T06:26:54.995ZINFOtest_disk_expunge (dropshot): request completed latency_us = 719 local_addr = [::1]:44559 method = GET remote_addr = [::1]:62034 req_id = acb789a9-351b-42c6-9590-ab9d88f432e5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
147712025-04-30T06:26:54.996ZDEBGtest_disk_expunge (BackgroundTasks): client response SledAgent = http://[::1]:44559 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:44559/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "acb789a9-351b-42c6-9590-ab9d88f432e5", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
147722025-04-30T06:26:55.001ZDEBGtest_disk_expunge (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
147732025-04-30T06:26:55.001ZDEBGtest_disk_expunge (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
147742025-04-30T06:26:55.001ZDEBGtest_disk_expunge (BackgroundTasks): finished collection background_task = inventory_collection
147752025-04-30T06:26:55.006ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
147762025-04-30T06:26:55.006ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
147772025-04-30T06:26:55.006ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
147782025-04-30T06:26:55.006ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
147792025-04-30T06:26:55.007ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
147802025-04-30T06:26:55.008ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
147812025-04-30T06:26:55.021ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
147822025-04-30T06:26:55.022ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147832025-04-30T06:26:55.022ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
147842025-04-30T06:26:55.023ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147852025-04-30T06:26:55.023ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 2 table_name = fields_string
147862025-04-30T06:26:55.023ZTRACtest_disk_expunge (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
147872025-04-30T06:26:55.027ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
147882025-04-30T06:26:55.031ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
147892025-04-30T06:26:55.031ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
147902025-04-30T06:26:55.031ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
147912025-04-30T06:26:55.032ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
147922025-04-30T06:26:55.033ZDEBGtest_disk_expunge (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = ea265a3d-3723-4b6c-bde0-f5fac0f54ad8 n_rows = 1 table_name = measurements_i64
147932025-04-30T06:26:55.034ZTRACtest_disk_expunge (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
147942025-04-30T06:26:55.034ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
147952025-04-30T06:26:55.034ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
147962025-04-30T06:26:55.035ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
147972025-04-30T06:26:55.042ZDEBGtest_disk_expunge (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
147982025-04-30T06:26:55.042ZDEBGtest_disk_expunge (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
147992025-04-30T06:26:55.050ZTRACtest_disk_expunge (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
148002025-04-30T06:26:55.052ZDEBGtest_disk_expunge (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
148012025-04-30T06:26:55.052ZTRACtest_disk_expunge (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
148022025-04-30T06:26:55.052ZTRACtest_disk_expunge (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
148032025-04-30T06:26:55.097ZWARNtest_disk_expunge (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
148042025-04-30T06:26:55.109ZWARNtest_disk_expunge (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
148052025-04-30T06:26:55.109ZWARNtest_disk_expunge (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
148062025-04-30T06:26:55.110ZDEBGtest_disk_expunge (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 501.445406ms iteration = 54 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
148072025-04-30T06:26:55.110ZDEBGtest_disk_expunge (BackgroundTasks): activating background_task = inventory_collection iteration = 55 reason = Signaled
148082025-04-30T06:26:55.110ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
148092025-04-30T06:26:55.110ZTRACtest_disk_expunge (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148102025-04-30T06:26:55.110ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148112025-04-30T06:26:55.110ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148122025-04-30T06:26:55.111ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148132025-04-30T06:26:55.114ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148142025-04-30T06:26:55.119ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
148152025-04-30T06:26:55.119ZTRACtest_disk_expunge (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
148162025-04-30T06:26:55.119ZDEBGtest_disk_expunge (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
148172025-04-30T06:26:55.119ZDEBGtest_disk_expunge (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
148182025-04-30T06:26:55.124ZDEBGtest_disk_expunge (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(5af26b40-5399-4991-864a-dbacb877f9ff, 0)]
148192025-04-30T06:26:55.124ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
148202025-04-30T06:26:55.132ZTRACtest_disk_expunge (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 7b e3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 {............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63699 req_id = 3f43aee6-84ae-4baa-8433-c7dfc0e8aef6
148212025-04-30T06:26:55.132ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31715,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63699 req_id = 3f43aee6-84ae-4baa-8433-c7dfc0e8aef6
148222025-04-30T06:26:55.132ZDEBGtest_disk_expunge (store): query key key = _mgs._tcp
148232025-04-30T06:26:55.133ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
148242025-04-30T06:26:55.133ZDEBGtest_disk_expunge (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63699 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3f43aee6-84ae-4baa-8433-c7dfc0e8aef6
148252025-04-30T06:26:55.133ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
148262025-04-30T06:26:55.134ZDEBGtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 41990, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3279, tv_nsec: 933312141 } })
148272025-04-30T06:26:55.134ZTRACtest_disk_expunge (DnsResolver): lookup_service_targets: looking up SRV target
148282025-04-30T06:26:55.135ZTRACtest_disk_expunge (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 66 8b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e f...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50608 req_id = 4ca32f39-9195-4a8d-b3a0-1147fb8a081e
148292025-04-30T06:26:55.135ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 26251,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50608 req_id = 4ca32f39-9195-4a8d-b3a0-1147fb8a081e
148302025-04-30T06:26:55.135ZDEBGtest_disk_expunge (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
148312025-04-30T06:26:55.136ZDEBGtest_disk_expunge (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50608 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4ca32f39-9195-4a8d-b3a0-1147fb8a081e
148322025-04-30T06:26:55.216ZTRACtest_disk_expunge (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
148332025-04-30T06:26:55.216ZTRACtest_disk_expunge (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
148342025-04-30T06:26:55.216ZTRACtest_disk_expunge (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
148352025-04-30T06:26:55.216ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148362025-04-30T06:26:55.216ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
148372025-04-30T06:26:55.217ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148382025-04-30T06:26:55.217ZTRACtest_disk_expunge (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 54 c8 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c T............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53674 req_id = 3e855a75-e03a-4f3b-9462-f6d42374316d
148392025-04-30T06:26:55.217ZDEBGtest_disk_expunge (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21704,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53674 req_id = 3e855a75-e03a-4f3b-9462-f6d42374316d
148402025-04-30T06:26:55.218ZDEBGtest_disk_expunge (store): query key key = _clickhouse-admin-keeper._tcp
148412025-04-30T06:26:55.218ZERROtest_disk_expunge (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21704, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53674 req_id = 3e855a75-e03a-4f3b-9462-f6d42374316d
148422025-04-30T06:26:55.218ZDEBGtest_disk_expunge (BackgroundTasks): begin collection background_task = inventory_collection
148432025-04-30T06:26:55.218ZDEBGtest_disk_expunge (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:41990
148442025-04-30T06:26:55.218ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/ignition
148452025-04-30T06:26:55.221ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3862f0f-1d39-4c0a-bfd5-f0ee9f2018ff", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148462025-04-30T06:26:55.221ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0
148472025-04-30T06:26:55.222ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
148482025-04-30T06:26:55.249ZDEBGtest_disk_expunge (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
148492025-04-30T06:26:55.249ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148502025-04-30T06:26:55.249ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
148512025-04-30T06:26:55.250ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148522025-04-30T06:26:55.252ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbaf69d6-2015-47b8-a418-5f9dbcc782bf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148532025-04-30T06:26:55.252ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0
148542025-04-30T06:26:55.253ZTRACtest_disk_expunge (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
148552025-04-30T06:26:55.253ZDEBGtest_disk_expunge (InternalApi): roles roles = RoleSet { roles: {} }
148562025-04-30T06:26:55.254ZDEBGtest_disk_expunge (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
148572025-04-30T06:26:55.262ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eea32094-0dc7-457b-b1c1-03bcd9261ff1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148582025-04-30T06:26:55.262ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1
148592025-04-30T06:26:55.274ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "44c4153b-9e76-4666-8465-8c12e8692cdc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148602025-04-30T06:26:55.276ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0
148612025-04-30T06:26:55.282ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cc3f1e6-cbff-47d4-a4bb-e9346d4eb4e1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148622025-04-30T06:26:55.283ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1
148632025-04-30T06:26:55.288ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b0ebab2-207a-4ae1-beb8-ea099f4e8c90", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148642025-04-30T06:26:55.288ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0
148652025-04-30T06:26:55.292ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b797e61c-e6a2-4ada-b67f-891f23918775", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148662025-04-30T06:26:55.292ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1
148672025-04-30T06:26:55.297ZDEBGtest_disk_expunge: attempt to fetch latest inventory collection result = Ok("found")
148682025-04-30T06:26:55.300ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df7fade9-5784-464f-bb59-e25b5a999bd5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148692025-04-30T06:26:55.301ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cmpa
148702025-04-30T06:26:55.303ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "650d386e-9719-46b2-a512-aa89e0f24ff3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148712025-04-30T06:26:55.303ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
148722025-04-30T06:26:55.304ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1f394c0-639e-4ba2-bf47-c7a4e638794b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148732025-04-30T06:26:55.305ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
148742025-04-30T06:26:55.306ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b754504-afff-4448-a71f-bfe3474e09d9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148752025-04-30T06:26:55.306ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/0/component/rot/cfpa
148762025-04-30T06:26:55.309ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc23d7e2-b5fe-4328-849e-fda98ef88e99", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148772025-04-30T06:26:55.309ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1
148782025-04-30T06:26:55.320ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ab3cc8f-01e5-4404-be35-3208954112e5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148792025-04-30T06:26:55.320ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0
148802025-04-30T06:26:55.324ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4ef8040-60e2-4ce3-b376-736a274f7e4e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148812025-04-30T06:26:55.325ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1
148822025-04-30T06:26:55.329ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c432023f-1231-48b0-a55e-485a22b3cbcd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148832025-04-30T06:26:55.329ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0
148842025-04-30T06:26:55.335ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f613597-3536-4384-a527-92b098041db4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148852025-04-30T06:26:55.335ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1
148862025-04-30T06:26:55.338ZDEBGtest_disk_expunge (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:41990 result = Ok(Response { url: "http://[::1]:41990/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c399a1d-b8ca-4e38-9b07-d0184c7e299a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:54 GMT"} })
148872025-04-30T06:26:55.339ZDEBGtest_disk_expunge (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:41990 method = GET uri = http://[::1]:41990/sp/switch/1/component/stage0/caboose?firmware_slot=0